Sep 11 16:16:45.963240 ip-10-0-143-197 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 16:16:46.358655 ip-10-0-143-197 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:46.358655 ip-10-0-143-197 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 16:16:46.358655 ip-10-0-143-197 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:46.359264 ip-10-0-143-197 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 16:16:46.359264 ip-10-0-143-197 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:46.359264 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.358755    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 16:16:46.362334 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362311    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:46.362334 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362333    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362338    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362341    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362345    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362348    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362351    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362354    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362357    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362359    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362362    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362365    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362368    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362371    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362373    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362376    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362379    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362381    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362384    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362387    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362389    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:46.362397 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362392    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362395    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362398    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362401    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362404    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362407    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362410    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362413    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362415    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362418    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362420    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362423    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362427    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362429    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362432    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362434    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362436    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362439    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362441    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362444    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:46.362979 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362446    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362449    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362451    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362453    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362456    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362458    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362461    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362464    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362468    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362471    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362474    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362477    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362479    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362483    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362485    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362488    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362491    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362493    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362496    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:46.363468 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362498    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362501    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362503    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362505    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362508    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362510    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362513    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362515    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362518    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362520    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362523    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362526    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362529    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362531    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362533    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362536    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362538    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362540    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362543    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362545    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:46.363928 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362547    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362550    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362554    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362558    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362560    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362563    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362566    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362569    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362573    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362576    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362579    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362582    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362585    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362587    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362589    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362592    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362595    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362597    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.362600    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:46.364460 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363312    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363318    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363321    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363324    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363327    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363330    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363333    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363336    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363339    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363343    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363345    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363348    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363350    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363353    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363355    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363358    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363361    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363363    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363365    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363369    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:46.364914 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363371    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363374    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363376    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363379    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363382    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363384    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363387    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363390    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363392    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363395    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363397    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363399    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363402    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363404    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363407    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363409    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363411    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363414    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363417    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363420    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:46.365433 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363422    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363425    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363428    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363430    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363433    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363435    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363438    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363440    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363445    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363448    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363451    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363455    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363458    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363461    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363463    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363466    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363468    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363471    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363473    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:46.365980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363476    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363478    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363481    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363484    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363486    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363489    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363491    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363493    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363496    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363499    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363501    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363504    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363507    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363509    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363512    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363515    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363518    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363520    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363522    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:46.366728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363525    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363529    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363532    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363535    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363538    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363542    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363545    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363547    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363550    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363552    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363555    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363557    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363560    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363562    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363565    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363567    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363570    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363572    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363574    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:46.367245 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363577    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363579    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363963    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363969    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363971    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363974    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363978    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363980    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363983    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363986    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363988    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363991    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363993    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363996    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.363998    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364001    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364003    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364006    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364011    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364014    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:46.367728 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364016    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364019    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364021    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364024    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364027    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364029    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364032    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364034    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364037    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364039    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364042    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364044    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364047    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364049    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364052    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364054    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364057    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364060    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364062    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364065    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:46.368239 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364068    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364070    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364072    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364075    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364077    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364080    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364082    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364084    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364087    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364089    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364093    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364096    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364099    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364102    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364104    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364106    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364109    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364111    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364114    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:46.368714 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364116    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364120    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364124    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364127    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364129    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364132    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364134    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364137    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364139    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364141    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364145    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364149    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364152    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364155    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364158    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364160    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364163    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364165    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364167    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:46.369207 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364170    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364172    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364175    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364177    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364181    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364183    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364186    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364188    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364191    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364193    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364197    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364200    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364203    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364206    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364209    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364211    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364213    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364216    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364218    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:46.369703 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364220    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364223    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364225    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364228    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364305    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364348    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364355    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364359    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364364    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364368    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364372    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364376    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364379    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364382    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364385    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364388    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364392    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364396    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364399    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364402    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364405    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364407    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364414    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 11 16:16:46.370174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364417    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364420    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364423    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364426    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364430    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364433    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364436    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364439    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364442    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364444    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364448    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364451    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364453    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364457    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364461    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364464    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364467    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364469    2576 flags.go:64] FLAG: --enable-server="true"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364473    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364477    2576 flags.go:64] FLAG: --event-burst="100"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364480    2576 flags.go:64] FLAG: --event-qps="50"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364483    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364485    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364488    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 11 16:16:46.370736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364492    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364495    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364498    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364501    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364504    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364507    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364510    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364512    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364515    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364518    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364520    2576 flags.go:64] FLAG: --feature-gates=""
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364524    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364527    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364530    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364533    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364535    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364540    2576 flags.go:64] FLAG: --help="false"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364542    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364545    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364548    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364551    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364554    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364557    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364560    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 16:16:46.371318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364563    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364565    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364568    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364571    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364574    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364577    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364579    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364582    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364585    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364588    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364591    2576 flags.go:64] FLAG: --lock-file=""
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364594    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364596    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364599    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364604    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364607    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364610    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364613    2576 flags.go:64] FLAG: --logging-format="text"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364616    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364619    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364621    2576 flags.go:64] FLAG: --manifest-url=""
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364624    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364628    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364631    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364636    2576 flags.go:64] FLAG: --max-pods="110"
Sep 11 16:16:46.371903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364639    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364642    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364645    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364648    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364650    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364653    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364656    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364662    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364665    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364668    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364671    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364674    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364679    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364682    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364686    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364688    2576 flags.go:64] FLAG: --port="10250"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364691    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364694    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040910940472ab8b6"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364697    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364700    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364703    2576 flags.go:64] FLAG: --register-node="true"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364706    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364709    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364712    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 11 16:16:46.372530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364715    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364717    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364725    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364728    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364731    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364733    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364736    2576 flags.go:64] FLAG: --runonce="false"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364740    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364744    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364746    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364749    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364752    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364755    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364758    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364761    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364764    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364767    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364770    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364772    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364776    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364779    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364782    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364787    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364790    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364792    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 16:16:46.373161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364797    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364800    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364802    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364805    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364808    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364810    2576 flags.go:64] FLAG: --v="2"
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364814    2576 flags.go:64] FLAG: --version="false"
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364818    2576 flags.go:64] FLAG: --vmodule=""
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364821    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.364824    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364966    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364970    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364974    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364976    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364981    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364983    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364986    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364990    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364993    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364996    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.364999    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:46.373757 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365002    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365005    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365008    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365011    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365013    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365016    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365018    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365022    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365024    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365027    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365029    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365033    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365036    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365038    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365040    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365043    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365045    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365048    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365050    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365053    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:46.374278 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365055    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365058    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365060    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365063    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365065    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365069    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365072    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365075    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365077    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365080    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365082    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365085    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365088    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365090    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365092    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365095    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365097    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365100    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365103    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365105    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:46.374772 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365108    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365111    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365113    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365117    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365119    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365122    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365124    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365127    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365129    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365132    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365134    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365137    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365139    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365142    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365144    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365148    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365152    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365156    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365158    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365161    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:46.375270 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365163    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365166    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365168    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365171    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365173    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365176    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365178    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365181    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365183    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365186    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365188    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365191    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365194    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365200    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365203    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365209    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365211    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365214    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365216    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:46.375758 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365219    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:46.376390 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365221    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:46.376390 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365224    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:46.376390 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365226    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:46.376390 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365229    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:46.376390 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365231    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:46.376390 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365234    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:46.376390 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365236    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:46.376390 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.365239    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:46.376390 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.366368    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:46.378393 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.378367    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 16:16:46.378393 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.378392    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 16:16:46.379152 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.379140    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 16:16:46.379212 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.379154    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 16:16:46.379266 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379253    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:46.379266 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379261    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:46.379266 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379266    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379271    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379276    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379281    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379286    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379290    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379294    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379298    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379302    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379306    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379321    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379326    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379331    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379335    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379339    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379343    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379347    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379357    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379361    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379366    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:46.379402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379370    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379382    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379386    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379390    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379394    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379398    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379402    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379406    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379410    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379415    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379419    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379422    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379426    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379430    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379434    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379438    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379442    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379446    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379450    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:46.380306 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379454    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379458    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379461    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379465    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379477    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379481    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379485    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379489    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379493    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379497    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379501    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379505    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379508    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379512    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379517    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379522    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379527    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379531    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379535    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:46.380993 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379539    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379543    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379547    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379551    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379554    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379558    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379562    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379567    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379571    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379575    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379579    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379583    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379587    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379591    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379595    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379599    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379602    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379608    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379621    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379626    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:46.381726 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379630    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379634    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379638    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379645    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379650    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379655    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379659    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379664    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379669    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379674    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379678    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379682    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379686    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379690    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379695    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379699    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379703    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379709    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:46.382602 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379715    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.379723    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379987    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.379998    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380004    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380009    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380014    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380019    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380023    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380028    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380032    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380036    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380040    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380051    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380056    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:46.383121 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380060    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380065    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380069    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380073    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380077    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380081    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380085    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380089    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380094    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380098    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380102    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380106    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380110    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380114    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380118    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380122    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380128    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380134    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380138    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380143    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:46.383661 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380147    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380151    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380156    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380160    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380164    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380167    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380171    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380175    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380179    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380183    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380187    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380199    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380203    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380207    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380212    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380215    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380219    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380223    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380228    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:46.384242 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380232    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380236    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380240    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380244    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380248    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380252    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380256    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380260    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380264    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380269    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380273    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380278    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380281    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380286    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380291    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380297    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380301    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380305    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380308    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380312    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:46.384765 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380316    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380320    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380323    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380327    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380331    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380343    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380347    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380351    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380355    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380359    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380363    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380367    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380371    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380376    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380379    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380384    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380388    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380392    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380396    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380400    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:46.385354 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380404    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380408    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380412    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380416    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380420    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380424    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:46.380428    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.380436    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.380606    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.383294    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.384319    2576 server.go:1019] "Starting client certificate rotation"
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.384421    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:16:46.385865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.385146    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:16:46.386911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.386895    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:16:46.391058 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.391036    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:16:46.401247 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.401228    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 11 16:16:46.405105 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.405091    2576 log.go:25] "Validated CRI v1 image API"
Sep 11 16:16:46.405927 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.405912    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 16:16:46.411730 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.411711    2576 fs.go:135] Filesystem UUIDs: map[4c43f028-f22c-4fcc-9349-e68861a2782e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b6037de9-2a65-4d0f-9f46-8140a8c8e3fa:/dev/nvme0n1p3]
Sep 11 16:16:46.411788 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.411730    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 16:16:46.417623 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.417503    2576 manager.go:216] Machine: {Timestamp:2026-09-11 16:16:46.4156985 +0000 UTC m=+0.325499652 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091943 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29eafc432b8bf4bb67a07c281a9a2a SystemUUID:ec29eafc-432b-8bf4-bb67-a07c281a9a2a BootID:b6d50b01-724e-4077-9b2c-222662422a21 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8a:b8:ad:5d:f7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8a:b8:ad:5d:f7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:1a:9c:eb:05:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 16:16:46.417623 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.417619    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 16:16:46.417751 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.417739    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 16:16:46.420572 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.420541    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 16:16:46.420716 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.420575    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-197.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 16:16:46.421257 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.421247    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 16:16:46.421289 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.421260    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 16:16:46.421289 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.421267    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:16:46.421289 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.421280    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:16:46.422020 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.422011    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 16:16:46.426446 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.426432    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:16:46.426564 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.426555    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 16:16:46.426607 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.426591    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 16:16:46.426607 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.426602    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 16:16:46.426661 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.426619    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 16:16:46.426661 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.426628    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 11 16:16:46.426661 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.426638    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 16:16:46.427058 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.427047    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:16:46.427099 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.427065    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:16:46.429058 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.429038    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 16:16:46.432081 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.432053    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 16:16:46.432180 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.432104    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 16:16:46.433565 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433552    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 16:16:46.433611 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433571    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 16:16:46.433611 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433581    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 16:16:46.433611 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433591    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 16:16:46.433611 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433601    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 16:16:46.433611 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433606    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 16:16:46.433744 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433620    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 16:16:46.433744 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433625    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 16:16:46.433744 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433631    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 16:16:46.433744 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433637    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 16:16:46.433744 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433655    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 16:16:46.433744 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433660    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 16:16:46.433912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.433821    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:16:46.434388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.434374    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 16:16:46.434419 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.434390    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 16:16:46.438633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.438620    2576 server.go:1268] "Started kubelet"
Sep 11 16:16:46.438738 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.438709    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 16:16:46.438789 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.438712    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 16:16:46.438846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.438788    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 16:16:46.439463 ip-10-0-143-197 systemd[1]: Started Kubernetes Kubelet.
Sep 11 16:16:46.440713 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.440647    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 16:16:46.441628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.441610    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-197.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:16:46.441703 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.441650    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:16:46.442012 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.441992    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:16:46.442276 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.442262    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 16:16:46.442969 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.442926    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 16:16:46.444408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.444389    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 16:16:46.445351 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.445338    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 16:16:46.445483 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.445464    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:46.445897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.445867    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 16:16:46.445988 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.445870    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 16:16:46.445988 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.445967    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 16:16:46.446156 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446133    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 16:16:46.446156 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446154    2576 factory.go:55] Registering systemd factory
Sep 11 16:16:46.446262 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446160    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 16:16:46.446262 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446168    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 16:16:46.446262 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446163    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 11 16:16:46.446501 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446488    2576 factory.go:153] Registering CRI-O factory
Sep 11 16:16:46.446563 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446505    2576 factory.go:223] Registration of the crio container factory successfully
Sep 11 16:16:46.446563 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446521    2576 factory.go:103] Registering Raw factory
Sep 11 16:16:46.446563 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446532    2576 manager.go:1196] Started watching for new ooms in manager
Sep 11 16:16:46.446825 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.446814    2576 manager.go:349] Starting recovery of all containers
Sep 11 16:16:46.450551 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.450496    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:16:46.451312 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.450277    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-197.ec2.internal.18d45072e9012c8b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-197.ec2.internal,UID:ip-10-0-143-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-197.ec2.internal,},FirstTimestamp:2026-09-11 16:16:46.438591627 +0000 UTC m=+0.348392781,LastTimestamp:2026-09-11 16:16:46.438591627 +0000 UTC m=+0.348392781,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-197.ec2.internal,}"
Sep 11 16:16:46.457469 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.457438    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 16:16:46.459317 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.459299    2576 manager.go:354] Recovery completed
Sep 11 16:16:46.464334 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.464323    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:46.465926 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.465903    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:46.466042 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.465979    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:46.466042 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.466000    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:46.466650 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.466631    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 16:16:46.466714 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.466651    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 16:16:46.466714 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.466636    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:46.466787 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.466668    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:16:46.467386 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.467372    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:46.467445 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.467399    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:46.467445 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.467412    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:46.468693 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.468632    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-197.ec2.internal.18d45072eaa247b8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-197.ec2.internal,UID:ip-10-0-143-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-197.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-197.ec2.internal,},FirstTimestamp:2026-09-11 16:16:46.465927096 +0000 UTC m=+0.375728252,LastTimestamp:2026-09-11 16:16:46.465927096 +0000 UTC m=+0.375728252,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-197.ec2.internal,}"
Sep 11 16:16:46.468927 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.468917    2576 policy_none.go:49] "None policy: Start"
Sep 11 16:16:46.468989 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.468949    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 16:16:46.468989 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.468960    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 16:16:46.471002 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.470992    2576 policy_none.go:47] "Start"
Sep 11 16:16:46.472005 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.471987    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2tpxb"
Sep 11 16:16:46.481313 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.481244    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-197.ec2.internal.18d45072eaa33ae4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-197.ec2.internal,UID:ip-10-0-143-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-197.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-197.ec2.internal,},FirstTimestamp:2026-09-11 16:16:46.465989348 +0000 UTC m=+0.375790505,LastTimestamp:2026-09-11 16:16:46.465989348 +0000 UTC m=+0.375790505,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-197.ec2.internal,}"
Sep 11 16:16:46.484620 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.484597    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2tpxb"
Sep 11 16:16:46.503586 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.503570    2576 manager.go:337] "Starting Device Plugin manager"
Sep 11 16:16:46.527390 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.503607    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 16:16:46.527390 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.503618    2576 server.go:87] "Starting device plugin registration server"
Sep 11 16:16:46.527390 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.503883    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 16:16:46.527390 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.503896    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 16:16:46.527390 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.503987    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 16:16:46.527390 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.504082    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 16:16:46.527390 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.504092    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 16:16:46.527390 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.504649    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 16:16:46.527390 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.504705    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:46.596013 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.595907    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 16:16:46.597258 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.597233    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 16:16:46.597333 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.597296    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 16:16:46.597333 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.597323    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 16:16:46.597410 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.597369    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 16:16:46.600526 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.600508    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:46.604380 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.604358    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:46.605164 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.605149    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:46.605263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.605182    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:46.605263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.605195    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:46.605263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.605223    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.615204 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.615189    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.615269 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.615210    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-197.ec2.internal\": node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:46.635835 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.635808    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:46.698174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.698140    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal"]
Sep 11 16:16:46.698283 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.698225    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:46.700012 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.699997    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:46.700100 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.700033    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:46.700100 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.700048    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:46.702131 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.702116    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:46.702255 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.702242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.702347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.702271    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:46.702790 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.702766    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:46.702790 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.702780    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:46.702916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.702796    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:46.702916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.702800    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:46.702916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.702808    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:46.702916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.702811    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:46.704815 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.704801    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.704874 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.704829    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:46.705478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.705462    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:46.705563 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.705485    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:46.705563 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.705494    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:46.736042 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.736022    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:46.744525 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.744507    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-197.ec2.internal\" not found" node="ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.748913 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.748896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52aec3f96d0d8edc36cd253742d0de5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal\" (UID: \"52aec3f96d0d8edc36cd253742d0de5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.749019 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.748918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83c5fb3c2cd6a94582ba9e97cbdde139-config\") pod \"kube-apiserver-proxy-ip-10-0-143-197.ec2.internal\" (UID: \"83c5fb3c2cd6a94582ba9e97cbdde139\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.749019 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.748951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52aec3f96d0d8edc36cd253742d0de5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal\" (UID: \"52aec3f96d0d8edc36cd253742d0de5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.749095 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.749033    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-197.ec2.internal\" not found" node="ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.836866 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.836834    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:46.849245 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.849182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52aec3f96d0d8edc36cd253742d0de5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal\" (UID: \"52aec3f96d0d8edc36cd253742d0de5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.849245 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.849217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83c5fb3c2cd6a94582ba9e97cbdde139-config\") pod \"kube-apiserver-proxy-ip-10-0-143-197.ec2.internal\" (UID: \"83c5fb3c2cd6a94582ba9e97cbdde139\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.849245 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.849238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52aec3f96d0d8edc36cd253742d0de5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal\" (UID: \"52aec3f96d0d8edc36cd253742d0de5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.849406 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.849272    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52aec3f96d0d8edc36cd253742d0de5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal\" (UID: \"52aec3f96d0d8edc36cd253742d0de5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.849406 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.849277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52aec3f96d0d8edc36cd253742d0de5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal\" (UID: \"52aec3f96d0d8edc36cd253742d0de5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.849406 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:46.849282    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83c5fb3c2cd6a94582ba9e97cbdde139-config\") pod \"kube-apiserver-proxy-ip-10-0-143-197.ec2.internal\" (UID: \"83c5fb3c2cd6a94582ba9e97cbdde139\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:46.937736 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:46.937709    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.038255 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.038235    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.046403 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.046386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:47.051966 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.051950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:47.138421 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.138348    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.238872 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.238841    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.339340 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.339299    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.383691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.383665    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 16:16:47.384113 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.383796    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:16:47.440221 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.440160    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.490199 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.490169    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 16:11:46 +0000 UTC" deadline="2028-06-22 14:38:10.046773494 +0000 UTC"
Sep 11 16:16:47.490199 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.490195    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15598h21m22.556581483s"
Sep 11 16:16:47.540462 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.540442    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.558880 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.558733    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:47.626357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.626335    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:47.627313 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.627291    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:47.627416 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.627331    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:47.627416 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.627345    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:47.641185 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.641162    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.660867 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:47.660843    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c5fb3c2cd6a94582ba9e97cbdde139.slice/crio-5ec20327fdb192df3cbda47898c1687406f2cecb0687e0b472bcc8b5c90ee867 WatchSource:0}: Error finding container 5ec20327fdb192df3cbda47898c1687406f2cecb0687e0b472bcc8b5c90ee867: Status 404 returned error can't find the container with id 5ec20327fdb192df3cbda47898c1687406f2cecb0687e0b472bcc8b5c90ee867
Sep 11 16:16:47.661568 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:47.661551    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52aec3f96d0d8edc36cd253742d0de5e.slice/crio-6d581fe76901be2cbbfa3bdfa8203e98762e1beb85138fb2d59aae9b92d14eee WatchSource:0}: Error finding container 6d581fe76901be2cbbfa3bdfa8203e98762e1beb85138fb2d59aae9b92d14eee: Status 404 returned error can't find the container with id 6d581fe76901be2cbbfa3bdfa8203e98762e1beb85138fb2d59aae9b92d14eee
Sep 11 16:16:47.665641 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.665628    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:16:47.741698 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.741626    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.847179 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.847140    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.947415 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:47.947366    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-197.ec2.internal\" not found"
Sep 11 16:16:47.965267 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.965241    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:47.995179 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:47.995115    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:48.046598 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.046568    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:48.064983 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.064957    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:16:48.065916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.065900    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal"
Sep 11 16:16:48.078914 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.078893    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:16:48.296763 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.296688    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:48.427899 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.427863    2576 apiserver.go:52] "Watching apiserver"
Sep 11 16:16:48.436368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.436339    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 16:16:48.436870 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.436840    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-8bwsp","openshift-multus/multus-additional-cni-plugins-7w7vl","openshift-network-diagnostics/network-check-target-dd2bv","openshift-network-operator/iptables-alerter-zkl78","openshift-ovn-kubernetes/ovnkube-node-gfkd5","kube-system/global-pull-secret-syncer-4gcmq","kube-system/konnectivity-agent-48rfb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf","openshift-cluster-node-tuning-operator/tuned-jrpsp","openshift-image-registry/node-ca-sjw2m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal","openshift-multus/multus-mn7vw","openshift-multus/network-metrics-daemon-9sg48","kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal"]
Sep 11 16:16:48.440281 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.439903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:16:48.442869 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.442840    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:48.442991 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.442912    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fjhqj\""
Sep 11 16:16:48.442991 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.442926    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:16:48.443195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.443178    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 16:16:48.443448 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.443419    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 16:16:48.445115 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.445087    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 16:16:48.445313 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.445288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:48.445394 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.445359    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:16:48.447625 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.447604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.451045 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.450597    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.451045 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.450603    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 16:16:48.451045 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.450686    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:16:48.451045 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.450605    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 16:16:48.451399 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.451302    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-48z7c\""
Sep 11 16:16:48.452826 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.452809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:48.452934 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.452869    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:16:48.455191 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.455143    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 16:16:48.455284 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.455261    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 16:16:48.455343 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.455318    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 16:16:48.460307 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.460286    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 16:16:48.460633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.460611    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l224q\""
Sep 11 16:16:48.461092 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.461075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.462279 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.462261    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.463505    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464201    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464436    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464462    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnxwp\" (UniqueName: \"kubernetes.io/projected/51c1c801-a06c-4efb-95ec-75ad310b2565-kube-api-access-hnxwp\") pod \"iptables-alerter-zkl78\" (UID: \"51c1c801-a06c-4efb-95ec-75ad310b2565\") " pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37d25eeb-883f-4742-a291-61e8a28f44fe-agent-certs\") pod \"konnectivity-agent-48rfb\" (UID: \"37d25eeb-883f-4742-a291-61e8a28f44fe\") " pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464595    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37d25eeb-883f-4742-a291-61e8a28f44fe-konnectivity-ca\") pod \"konnectivity-agent-48rfb\" (UID: \"37d25eeb-883f-4742-a291-61e8a28f44fe\") " pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464713    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvf2s\" (UniqueName: \"kubernetes.io/projected/e45bda4f-12ee-46a5-8d83-7a1602904679-kube-api-access-wvf2s\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51c1c801-a06c-4efb-95ec-75ad310b2565-iptables-alerter-script\") pod \"iptables-alerter-zkl78\" (UID: \"51c1c801-a06c-4efb-95ec-75ad310b2565\") " pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.464959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.464778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51c1c801-a06c-4efb-95ec-75ad310b2565-host-slash\") pod \"iptables-alerter-zkl78\" (UID: \"51c1c801-a06c-4efb-95ec-75ad310b2565\") " pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.465628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.465406    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-db5rw\""
Sep 11 16:16:48.474719 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.474697    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 16:16:48.474999 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.474968    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 16:16:48.475209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.475192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.475311 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.475208    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:16:48.475464 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.475435    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 16:16:48.476225 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.475794    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g8sbm\""
Sep 11 16:16:48.476225 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.475838    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:16:48.476225 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.475945    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 16:16:48.476225 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.476105    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 16:16:48.476225 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.476134    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlqpv\""
Sep 11 16:16:48.476225 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.476185    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 16:16:48.476572 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.476349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.480218 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.480197    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 16:16:48.480381 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.480362    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.480522 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.480507    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 16:16:48.481099 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.481079    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 16:16:48.481201 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.481101    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 16:16:48.481201 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.481143    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5w64d\""
Sep 11 16:16:48.481385 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.481367    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 16:16:48.481439 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.481417    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6mnjx\""
Sep 11 16:16:48.481509 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.481492    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 16:16:48.486463 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.486445    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 16:16:48.486885 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.486866    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g2qss\""
Sep 11 16:16:48.499483 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.499461    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jq9gz"
Sep 11 16:16:48.512596 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.512558    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jq9gz"
Sep 11 16:16:48.547289 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.547114    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 16:16:48.564914 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.564884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:48.565044 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.564917    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-socket-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.565044 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.564934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db333929-74ef-449c-a79a-6878ef6051c1-tmp-dir\") pod \"node-resolver-8bwsp\" (UID: \"db333929-74ef-449c-a79a-6878ef6051c1\") " pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.565044 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.564971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-run-k8s-cni-cncf-io\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.565044 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.564998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74c8f10b-e859-4c3e-b95a-5f58ee099241-kubelet-config\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:48.565044 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-sysctl-d\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvf2s\" (UniqueName: \"kubernetes.io/projected/e45bda4f-12ee-46a5-8d83-7a1602904679-kube-api-access-wvf2s\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51c1c801-a06c-4efb-95ec-75ad310b2565-iptables-alerter-script\") pod \"iptables-alerter-zkl78\" (UID: \"51c1c801-a06c-4efb-95ec-75ad310b2565\") " pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565096    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh4j8\" (UniqueName: \"kubernetes.io/projected/a9648594-e0ca-46fd-ab95-86cfaa507bae-kube-api-access-vh4j8\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg8dr\" (UniqueName: \"kubernetes.io/projected/5e2a6297-0738-43c3-9d88-7cfa8d9474d7-kube-api-access-lg8dr\") pod \"node-ca-sjw2m\" (UID: \"5e2a6297-0738-43c3-9d88-7cfa8d9474d7\") " pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-registration-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-sys-fs\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565174    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c911c447-3778-4e0c-bd78-297c361a8740-ovnkube-config\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-run-netns\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-tuned\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-cni-binary-copy\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37d25eeb-883f-4742-a291-61e8a28f44fe-agent-certs\") pod \"konnectivity-agent-48rfb\" (UID: \"37d25eeb-883f-4742-a291-61e8a28f44fe\") " pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:16:48.565314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9648594-e0ca-46fd-ab95-86cfaa507bae-cnibin\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9648594-e0ca-46fd-ab95-86cfaa507bae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565379    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-kubelet\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-run-openvswitch\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-run\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxllh\" (UniqueName: \"kubernetes.io/projected/e725153a-43ec-44b9-a9bc-53ee55889dc9-kube-api-access-cxllh\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-cni-dir\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-run-systemd\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565522    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-cni-bin\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-systemd\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565572    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-lib-modules\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e2a6297-0738-43c3-9d88-7cfa8d9474d7-host\") pod \"node-ca-sjw2m\" (UID: \"5e2a6297-0738-43c3-9d88-7cfa8d9474d7\") " pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e2a6297-0738-43c3-9d88-7cfa8d9474d7-serviceca\") pod \"node-ca-sjw2m\" (UID: \"5e2a6297-0738-43c3-9d88-7cfa8d9474d7\") " pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/db333929-74ef-449c-a79a-6878ef6051c1-hosts-file\") pod \"node-resolver-8bwsp\" (UID: \"db333929-74ef-449c-a79a-6878ef6051c1\") " pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-var-lib-kubelet\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565665    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9648594-e0ca-46fd-ab95-86cfaa507bae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.566006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-log-socket\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-run-ovn-kubernetes\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-conf-dir\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c911c447-3778-4e0c-bd78-297c361a8740-ovnkube-script-lib\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2rgz\" (UniqueName: \"kubernetes.io/projected/c911c447-3778-4e0c-bd78-297c361a8740-kube-api-access-t2rgz\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e725153a-43ec-44b9-a9bc-53ee55889dc9-tmp\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-run-netns\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-cni-netd\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-hostroot\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.565976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-device-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c911c447-3778-4e0c-bd78-297c361a8740-env-overrides\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-daemon-config\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74c8f10b-e859-4c3e-b95a-5f58ee099241-dbus\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-sysconfig\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-var-lib-kubelet\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.566857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-etc-openvswitch\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr4jp\" (UniqueName: \"kubernetes.io/projected/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-kube-api-access-sr4jp\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-modprobe-d\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-host\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51c1c801-a06c-4efb-95ec-75ad310b2565-host-slash\") pod \"iptables-alerter-zkl78\" (UID: \"51c1c801-a06c-4efb-95ec-75ad310b2565\") " pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-system-cni-dir\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-run-multus-certs\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-socket-dir-parent\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37d25eeb-883f-4742-a291-61e8a28f44fe-konnectivity-ca\") pod \"konnectivity-agent-48rfb\" (UID: \"37d25eeb-883f-4742-a291-61e8a28f44fe\") " pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9648594-e0ca-46fd-ab95-86cfaa507bae-system-cni-dir\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4j85\" (UniqueName: \"kubernetes.io/projected/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-kube-api-access-w4j85\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-slash\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-var-lib-openvswitch\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c911c447-3778-4e0c-bd78-297c361a8740-ovn-node-metrics-cert\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.567699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-cnibin\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxwp\" (UniqueName: \"kubernetes.io/projected/51c1c801-a06c-4efb-95ec-75ad310b2565-kube-api-access-hnxwp\") pod \"iptables-alerter-zkl78\" (UID: \"51c1c801-a06c-4efb-95ec-75ad310b2565\") " pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9648594-e0ca-46fd-ab95-86cfaa507bae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-systemd-units\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wbv\" (UniqueName: \"kubernetes.io/projected/db333929-74ef-449c-a79a-6878ef6051c1-kube-api-access-h6wbv\") pod \"node-resolver-8bwsp\" (UID: \"db333929-74ef-449c-a79a-6878ef6051c1\") " pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-var-lib-cni-bin\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-etc-kubernetes\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9648594-e0ca-46fd-ab95-86cfaa507bae-cni-binary-copy\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-run-ovn\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.566998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-sysctl-conf\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.567051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-sys\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.567077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-node-log\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.567105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51c1c801-a06c-4efb-95ec-75ad310b2565-iptables-alerter-script\") pod \"iptables-alerter-zkl78\" (UID: \"51c1c801-a06c-4efb-95ec-75ad310b2565\") " pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.567105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-var-lib-cni-multus\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.567295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-kubernetes\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.568585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.567731    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51c1c801-a06c-4efb-95ec-75ad310b2565-host-slash\") pod \"iptables-alerter-zkl78\" (UID: \"51c1c801-a06c-4efb-95ec-75ad310b2565\") " pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.573633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.567834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37d25eeb-883f-4742-a291-61e8a28f44fe-konnectivity-ca\") pod \"konnectivity-agent-48rfb\" (UID: \"37d25eeb-883f-4742-a291-61e8a28f44fe\") " pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:16:48.573633 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.567930    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:48.573633 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.568460    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs podName:e45bda4f-12ee-46a5-8d83-7a1602904679 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:49.068428666 +0000 UTC m=+2.978229825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs") pod "network-metrics-daemon-9sg48" (UID: "e45bda4f-12ee-46a5-8d83-7a1602904679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:48.573633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.572388    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37d25eeb-883f-4742-a291-61e8a28f44fe-agent-certs\") pod \"konnectivity-agent-48rfb\" (UID: \"37d25eeb-883f-4742-a291-61e8a28f44fe\") " pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:16:48.581321 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.581299    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:48.581424 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.581331    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:48.581424 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.581346    2576 projected.go:196] Error preparing data for projected volume kube-api-access-9prrq for pod openshift-network-diagnostics/network-check-target-dd2bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:48.581424 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.581402    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq podName:df08f2ff-be81-4cc3-966d-bc1be8960c4a nodeName:}" failed. No retries permitted until 2026-09-11 16:16:49.081383745 +0000 UTC m=+2.991184892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9prrq" (UniqueName: "kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq") pod "network-check-target-dd2bv" (UID: "df08f2ff-be81-4cc3-966d-bc1be8960c4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:48.582325 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.582304    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvf2s\" (UniqueName: \"kubernetes.io/projected/e45bda4f-12ee-46a5-8d83-7a1602904679-kube-api-access-wvf2s\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:48.587978 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.587928    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxwp\" (UniqueName: \"kubernetes.io/projected/51c1c801-a06c-4efb-95ec-75ad310b2565-kube-api-access-hnxwp\") pod \"iptables-alerter-zkl78\" (UID: \"51c1c801-a06c-4efb-95ec-75ad310b2565\") " pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.605438 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.605390    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal" event={"ID":"52aec3f96d0d8edc36cd253742d0de5e","Type":"ContainerStarted","Data":"6d581fe76901be2cbbfa3bdfa8203e98762e1beb85138fb2d59aae9b92d14eee"}
Sep 11 16:16:48.607396 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.607368    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal" event={"ID":"83c5fb3c2cd6a94582ba9e97cbdde139","Type":"ContainerStarted","Data":"5ec20327fdb192df3cbda47898c1687406f2cecb0687e0b472bcc8b5c90ee867"}
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-socket-dir-parent\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9648594-e0ca-46fd-ab95-86cfaa507bae-system-cni-dir\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668170    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4j85\" (UniqueName: \"kubernetes.io/projected/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-kube-api-access-w4j85\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-slash\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-var-lib-openvswitch\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c911c447-3778-4e0c-bd78-297c361a8740-ovn-node-metrics-cert\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-cnibin\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9648594-e0ca-46fd-ab95-86cfaa507bae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-systemd-units\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wbv\" (UniqueName: \"kubernetes.io/projected/db333929-74ef-449c-a79a-6878ef6051c1-kube-api-access-h6wbv\") pod \"node-resolver-8bwsp\" (UID: \"db333929-74ef-449c-a79a-6878ef6051c1\") " pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-var-lib-cni-bin\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-etc-kubernetes\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9648594-e0ca-46fd-ab95-86cfaa507bae-cni-binary-copy\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668513    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-run-ovn\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.671022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-sysctl-conf\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-sys\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-node-log\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-var-lib-cni-multus\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-kubernetes\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-socket-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db333929-74ef-449c-a79a-6878ef6051c1-tmp-dir\") pod \"node-resolver-8bwsp\" (UID: \"db333929-74ef-449c-a79a-6878ef6051c1\") " pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-run-k8s-cni-cncf-io\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74c8f10b-e859-4c3e-b95a-5f58ee099241-kubelet-config\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-sysctl-d\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh4j8\" (UniqueName: \"kubernetes.io/projected/a9648594-e0ca-46fd-ab95-86cfaa507bae-kube-api-access-vh4j8\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg8dr\" (UniqueName: \"kubernetes.io/projected/5e2a6297-0738-43c3-9d88-7cfa8d9474d7-kube-api-access-lg8dr\") pod \"node-ca-sjw2m\" (UID: \"5e2a6297-0738-43c3-9d88-7cfa8d9474d7\") " pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-registration-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668917    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-sys-fs\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.668980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c911c447-3778-4e0c-bd78-297c361a8740-ovnkube-config\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-run-netns\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.671972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-tuned\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-cni-binary-copy\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9648594-e0ca-46fd-ab95-86cfaa507bae-cnibin\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9648594-e0ca-46fd-ab95-86cfaa507bae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669170    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-kubelet\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-run-openvswitch\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669223    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-run\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxllh\" (UniqueName: \"kubernetes.io/projected/e725153a-43ec-44b9-a9bc-53ee55889dc9-kube-api-access-cxllh\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-cni-dir\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-run-systemd\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-cni-bin\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669377    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-systemd\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-lib-modules\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e2a6297-0738-43c3-9d88-7cfa8d9474d7-host\") pod \"node-ca-sjw2m\" (UID: \"5e2a6297-0738-43c3-9d88-7cfa8d9474d7\") " pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e2a6297-0738-43c3-9d88-7cfa8d9474d7-serviceca\") pod \"node-ca-sjw2m\" (UID: \"5e2a6297-0738-43c3-9d88-7cfa8d9474d7\") " pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669488    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/db333929-74ef-449c-a79a-6878ef6051c1-hosts-file\") pod \"node-resolver-8bwsp\" (UID: \"db333929-74ef-449c-a79a-6878ef6051c1\") " pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-var-lib-kubelet\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669542    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9648594-e0ca-46fd-ab95-86cfaa507bae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.673043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-log-socket\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-run-ovn-kubernetes\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-conf-dir\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c911c447-3778-4e0c-bd78-297c361a8740-ovnkube-script-lib\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2rgz\" (UniqueName: \"kubernetes.io/projected/c911c447-3778-4e0c-bd78-297c361a8740-kube-api-access-t2rgz\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e725153a-43ec-44b9-a9bc-53ee55889dc9-tmp\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-run-netns\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-cni-netd\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-hostroot\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-device-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c911c447-3778-4e0c-bd78-297c361a8740-env-overrides\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-daemon-config\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74c8f10b-e859-4c3e-b95a-5f58ee099241-dbus\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-sysconfig\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.669981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-var-lib-kubelet\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-etc-openvswitch\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.674095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr4jp\" (UniqueName: \"kubernetes.io/projected/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-kube-api-access-sr4jp\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-modprobe-d\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-host\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-system-cni-dir\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-run-multus-certs\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670252    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-run-multus-certs\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-sysctl-conf\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-cni-binary-copy\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670666    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-socket-dir-parent\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670701    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9648594-e0ca-46fd-ab95-86cfaa507bae-system-cni-dir\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670705    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9648594-e0ca-46fd-ab95-86cfaa507bae-cnibin\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670790    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-sys\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-systemd-units\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-slash\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.670937    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-var-lib-openvswitch\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9648594-e0ca-46fd-ab95-86cfaa507bae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671148    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-sysctl-d\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.675206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671153    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-kubelet\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671203    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-node-log\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-run-openvswitch\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671248    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-var-lib-cni-multus\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671270    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-run\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-kubernetes\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-cni-dir\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db333929-74ef-449c-a79a-6878ef6051c1-tmp-dir\") pod \"node-resolver-8bwsp\" (UID: \"db333929-74ef-449c-a79a-6878ef6051c1\") " pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671614    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-cni-bin\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671664    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-systemd\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-run-k8s-cni-cncf-io\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671718    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-registration-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-lib-modules\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.671848    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:48.671899    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret podName:74c8f10b-e859-4c3e-b95a-5f58ee099241 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:49.171882816 +0000 UTC m=+3.081683960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret") pod "global-pull-secret-syncer-4gcmq" (UID: "74c8f10b-e859-4c3e-b95a-5f58ee099241") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672028    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74c8f10b-e859-4c3e-b95a-5f58ee099241-kubelet-config\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672073    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e2a6297-0738-43c3-9d88-7cfa8d9474d7-host\") pod \"node-ca-sjw2m\" (UID: \"5e2a6297-0738-43c3-9d88-7cfa8d9474d7\") " pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.671770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-sys-fs\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.676261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672129    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-var-lib-cni-bin\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672168    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e2a6297-0738-43c3-9d88-7cfa8d9474d7-serviceca\") pod \"node-ca-sjw2m\" (UID: \"5e2a6297-0738-43c3-9d88-7cfa8d9474d7\") " pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/db333929-74ef-449c-a79a-6878ef6051c1-hosts-file\") pod \"node-resolver-8bwsp\" (UID: \"db333929-74ef-449c-a79a-6878ef6051c1\") " pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672267    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-var-lib-kubelet\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672321    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9648594-e0ca-46fd-ab95-86cfaa507bae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672379    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-log-socket\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672396    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c911c447-3778-4e0c-bd78-297c361a8740-ovnkube-config\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-run-ovn-kubernetes\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672450    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-host-run-netns\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672487    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-run-systemd\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672532    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-run-netns\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672575    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-conf-dir\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c911c447-3778-4e0c-bd78-297c361a8740-env-overrides\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672848    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-host-cni-netd\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672892    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-hostroot\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672921    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-multus-daemon-config\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672944    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.677209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.672996    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-device-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673001    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74c8f10b-e859-4c3e-b95a-5f58ee099241-dbus\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673098    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-sysconfig\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673108    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c911c447-3778-4e0c-bd78-297c361a8740-ovnkube-script-lib\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673164    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-cnibin\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-etc-openvswitch\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-var-lib-kubelet\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673337    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-system-cni-dir\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673341    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-modprobe-d\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673391    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e725153a-43ec-44b9-a9bc-53ee55889dc9-host\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673461    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-socket-dir\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673460    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e725153a-43ec-44b9-a9bc-53ee55889dc9-etc-tuned\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673599    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c911c447-3778-4e0c-bd78-297c361a8740-run-ovn\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673669    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-etc-kubernetes\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.673989    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9648594-e0ca-46fd-ab95-86cfaa507bae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.675630    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9648594-e0ca-46fd-ab95-86cfaa507bae-cni-binary-copy\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.676628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e725153a-43ec-44b9-a9bc-53ee55889dc9-tmp\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.677839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.677452    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c911c447-3778-4e0c-bd78-297c361a8740-ovn-node-metrics-cert\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.695063 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.695041    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr4jp\" (UniqueName: \"kubernetes.io/projected/ea980b3b-40fd-4ea0-a4b3-70706ab8c942-kube-api-access-sr4jp\") pod \"multus-mn7vw\" (UID: \"ea980b3b-40fd-4ea0-a4b3-70706ab8c942\") " pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.699967 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.699926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2rgz\" (UniqueName: \"kubernetes.io/projected/c911c447-3778-4e0c-bd78-297c361a8740-kube-api-access-t2rgz\") pod \"ovnkube-node-gfkd5\" (UID: \"c911c447-3778-4e0c-bd78-297c361a8740\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.700373 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.700339    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh4j8\" (UniqueName: \"kubernetes.io/projected/a9648594-e0ca-46fd-ab95-86cfaa507bae-kube-api-access-vh4j8\") pod \"multus-additional-cni-plugins-7w7vl\" (UID: \"a9648594-e0ca-46fd-ab95-86cfaa507bae\") " pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.700470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.700350    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxllh\" (UniqueName: \"kubernetes.io/projected/e725153a-43ec-44b9-a9bc-53ee55889dc9-kube-api-access-cxllh\") pod \"tuned-jrpsp\" (UID: \"e725153a-43ec-44b9-a9bc-53ee55889dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.700470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.700444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg8dr\" (UniqueName: \"kubernetes.io/projected/5e2a6297-0738-43c3-9d88-7cfa8d9474d7-kube-api-access-lg8dr\") pod \"node-ca-sjw2m\" (UID: \"5e2a6297-0738-43c3-9d88-7cfa8d9474d7\") " pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.700837 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.700814    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4j85\" (UniqueName: \"kubernetes.io/projected/8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399-kube-api-access-w4j85\") pod \"aws-ebs-csi-driver-node-vzvqf\" (UID: \"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.700912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.700824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wbv\" (UniqueName: \"kubernetes.io/projected/db333929-74ef-449c-a79a-6878ef6051c1-kube-api-access-h6wbv\") pod \"node-resolver-8bwsp\" (UID: \"db333929-74ef-449c-a79a-6878ef6051c1\") " pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.756232 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.756208    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:16:48.766504 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:48.766473    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d25eeb_883f_4742_a291_61e8a28f44fe.slice/crio-a2f18e1b4423a4f4066f92d0a817d52013cd6104b54b834612363d615d1c5810 WatchSource:0}: Error finding container a2f18e1b4423a4f4066f92d0a817d52013cd6104b54b834612363d615d1c5810: Status 404 returned error can't find the container with id a2f18e1b4423a4f4066f92d0a817d52013cd6104b54b834612363d615d1c5810
Sep 11 16:16:48.781246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.781220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zkl78"
Sep 11 16:16:48.791044 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:48.790998    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c1c801_a06c_4efb_95ec_75ad310b2565.slice/crio-998b743b347d94b6cf7882646b25d88aee0776d1b1ced4d7ba3e83b5c293a3d9 WatchSource:0}: Error finding container 998b743b347d94b6cf7882646b25d88aee0776d1b1ced4d7ba3e83b5c293a3d9: Status 404 returned error can't find the container with id 998b743b347d94b6cf7882646b25d88aee0776d1b1ced4d7ba3e83b5c293a3d9
Sep 11 16:16:48.792887 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.792867    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:16:48.799865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.799800    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8bwsp"
Sep 11 16:16:48.803835 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:48.803812    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc911c447_3778_4e0c_bd78_297c361a8740.slice/crio-21ff05bdc0a75f45145692fe02ba85e1c24a66462a487dd0fbdc5c8e863e0fdc WatchSource:0}: Error finding container 21ff05bdc0a75f45145692fe02ba85e1c24a66462a487dd0fbdc5c8e863e0fdc: Status 404 returned error can't find the container with id 21ff05bdc0a75f45145692fe02ba85e1c24a66462a487dd0fbdc5c8e863e0fdc
Sep 11 16:16:48.810004 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:48.809981    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb333929_74ef_449c_a79a_6878ef6051c1.slice/crio-b7fe9d4dc0c7886f73269224ec28688b085c61a40ffe55f650ae99f3b92d6962 WatchSource:0}: Error finding container b7fe9d4dc0c7886f73269224ec28688b085c61a40ffe55f650ae99f3b92d6962: Status 404 returned error can't find the container with id b7fe9d4dc0c7886f73269224ec28688b085c61a40ffe55f650ae99f3b92d6962
Sep 11 16:16:48.815504 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.815482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jrpsp"
Sep 11 16:16:48.824812 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.824782    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7w7vl"
Sep 11 16:16:48.826822 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:48.826788    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode725153a_43ec_44b9_a9bc_53ee55889dc9.slice/crio-b2b15baf7bc00dacee10de8f907911bce2c95f5eee4a0ec1d82161494741599c WatchSource:0}: Error finding container b2b15baf7bc00dacee10de8f907911bce2c95f5eee4a0ec1d82161494741599c: Status 404 returned error can't find the container with id b2b15baf7bc00dacee10de8f907911bce2c95f5eee4a0ec1d82161494741599c
Sep 11 16:16:48.833663 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.833642    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sjw2m"
Sep 11 16:16:48.836652 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:48.836629    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9648594_e0ca_46fd_ab95_86cfaa507bae.slice/crio-0fb07541c846df69da91d6886f7a5a124666a35795be32971431fc8cf241a83e WatchSource:0}: Error finding container 0fb07541c846df69da91d6886f7a5a124666a35795be32971431fc8cf241a83e: Status 404 returned error can't find the container with id 0fb07541c846df69da91d6886f7a5a124666a35795be32971431fc8cf241a83e
Sep 11 16:16:48.842740 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.842721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf"
Sep 11 16:16:48.844325 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:48.844302    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2a6297_0738_43c3_9d88_7cfa8d9474d7.slice/crio-ab74e88ed46d495a1e4ccebdec58eed0f3013b470c98a48e905ab6ced14332a2 WatchSource:0}: Error finding container ab74e88ed46d495a1e4ccebdec58eed0f3013b470c98a48e905ab6ced14332a2: Status 404 returned error can't find the container with id ab74e88ed46d495a1e4ccebdec58eed0f3013b470c98a48e905ab6ced14332a2
Sep 11 16:16:48.852003 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:48.851979    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mn7vw"
Sep 11 16:16:48.854573 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:48.854547    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7a6cd3_e0a9_4dd0_ad13_be32f0b2d399.slice/crio-06549ec58cdfd2d4e31827b5a68e2e6019b56593251eaad67e1ff146e6b3b08d WatchSource:0}: Error finding container 06549ec58cdfd2d4e31827b5a68e2e6019b56593251eaad67e1ff146e6b3b08d: Status 404 returned error can't find the container with id 06549ec58cdfd2d4e31827b5a68e2e6019b56593251eaad67e1ff146e6b3b08d
Sep 11 16:16:48.862402 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:16:48.862379    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea980b3b_40fd_4ea0_a4b3_70706ab8c942.slice/crio-c5ab4d9988956d6b2db2634716038f0e15cad83d62609c549ad4bd9eb6b1377c WatchSource:0}: Error finding container c5ab4d9988956d6b2db2634716038f0e15cad83d62609c549ad4bd9eb6b1377c: Status 404 returned error can't find the container with id c5ab4d9988956d6b2db2634716038f0e15cad83d62609c549ad4bd9eb6b1377c
Sep 11 16:16:49.074198 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.074084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:49.074354 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:49.074261    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:49.074354 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:49.074342    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs podName:e45bda4f-12ee-46a5-8d83-7a1602904679 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:50.07431978 +0000 UTC m=+3.984120925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs") pod "network-metrics-daemon-9sg48" (UID: "e45bda4f-12ee-46a5-8d83-7a1602904679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:49.174771 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.174725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:49.174771 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.174769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:49.175034 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:49.174867    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:49.175034 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:49.174900    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:49.175034 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:49.174925    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:49.175034 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:49.174940    2576 projected.go:196] Error preparing data for projected volume kube-api-access-9prrq for pod openshift-network-diagnostics/network-check-target-dd2bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:49.175034 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:49.174925    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret podName:74c8f10b-e859-4c3e-b95a-5f58ee099241 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:50.17491187 +0000 UTC m=+4.084713013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret") pod "global-pull-secret-syncer-4gcmq" (UID: "74c8f10b-e859-4c3e-b95a-5f58ee099241") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:49.175286 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:49.175039    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq podName:df08f2ff-be81-4cc3-966d-bc1be8960c4a nodeName:}" failed. No retries permitted until 2026-09-11 16:16:50.175020911 +0000 UTC m=+4.084822066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9prrq" (UniqueName: "kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq") pod "network-check-target-dd2bv" (UID: "df08f2ff-be81-4cc3-966d-bc1be8960c4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:49.516694 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.516620    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:11:48 +0000 UTC" deadline="2028-03-16 12:29:50.015434129 +0000 UTC"
Sep 11 16:16:49.516694 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.516686    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13244h13m0.498752851s"
Sep 11 16:16:49.598097 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.598062    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:49.598286 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:49.598201    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:16:49.611717 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.611377    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7w7vl" event={"ID":"a9648594-e0ca-46fd-ab95-86cfaa507bae","Type":"ContainerStarted","Data":"0fb07541c846df69da91d6886f7a5a124666a35795be32971431fc8cf241a83e"}
Sep 11 16:16:49.613367 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.613243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jrpsp" event={"ID":"e725153a-43ec-44b9-a9bc-53ee55889dc9","Type":"ContainerStarted","Data":"b2b15baf7bc00dacee10de8f907911bce2c95f5eee4a0ec1d82161494741599c"}
Sep 11 16:16:49.615449 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.615421    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerStarted","Data":"21ff05bdc0a75f45145692fe02ba85e1c24a66462a487dd0fbdc5c8e863e0fdc"}
Sep 11 16:16:49.617547 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.617522    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-48rfb" event={"ID":"37d25eeb-883f-4742-a291-61e8a28f44fe","Type":"ContainerStarted","Data":"a2f18e1b4423a4f4066f92d0a817d52013cd6104b54b834612363d615d1c5810"}
Sep 11 16:16:49.619202 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.619168    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mn7vw" event={"ID":"ea980b3b-40fd-4ea0-a4b3-70706ab8c942","Type":"ContainerStarted","Data":"c5ab4d9988956d6b2db2634716038f0e15cad83d62609c549ad4bd9eb6b1377c"}
Sep 11 16:16:49.620912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.620889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf" event={"ID":"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399","Type":"ContainerStarted","Data":"06549ec58cdfd2d4e31827b5a68e2e6019b56593251eaad67e1ff146e6b3b08d"}
Sep 11 16:16:49.622384 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.622362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sjw2m" event={"ID":"5e2a6297-0738-43c3-9d88-7cfa8d9474d7","Type":"ContainerStarted","Data":"ab74e88ed46d495a1e4ccebdec58eed0f3013b470c98a48e905ab6ced14332a2"}
Sep 11 16:16:49.624194 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.624165    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8bwsp" event={"ID":"db333929-74ef-449c-a79a-6878ef6051c1","Type":"ContainerStarted","Data":"b7fe9d4dc0c7886f73269224ec28688b085c61a40ffe55f650ae99f3b92d6962"}
Sep 11 16:16:49.625577 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:49.625550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zkl78" event={"ID":"51c1c801-a06c-4efb-95ec-75ad310b2565","Type":"ContainerStarted","Data":"998b743b347d94b6cf7882646b25d88aee0776d1b1ced4d7ba3e83b5c293a3d9"}
Sep 11 16:16:50.081912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:50.081795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:50.082075 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.082003    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:50.082144 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.082086    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs podName:e45bda4f-12ee-46a5-8d83-7a1602904679 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:52.082064306 +0000 UTC m=+5.991865453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs") pod "network-metrics-daemon-9sg48" (UID: "e45bda4f-12ee-46a5-8d83-7a1602904679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:50.183234 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:50.183193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:50.183468 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:50.183255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:50.183468 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.183396    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:50.183468 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.183465    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret podName:74c8f10b-e859-4c3e-b95a-5f58ee099241 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:52.183446057 +0000 UTC m=+6.093247204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret") pod "global-pull-secret-syncer-4gcmq" (UID: "74c8f10b-e859-4c3e-b95a-5f58ee099241") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:50.183970 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.183951    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:50.184067 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.183976    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:50.184067 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.183999    2576 projected.go:196] Error preparing data for projected volume kube-api-access-9prrq for pod openshift-network-diagnostics/network-check-target-dd2bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:50.184219 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.184069    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq podName:df08f2ff-be81-4cc3-966d-bc1be8960c4a nodeName:}" failed. No retries permitted until 2026-09-11 16:16:52.184051146 +0000 UTC m=+6.093852292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9prrq" (UniqueName: "kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq") pod "network-check-target-dd2bv" (UID: "df08f2ff-be81-4cc3-966d-bc1be8960c4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:50.517196 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:50.517067    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:11:48 +0000 UTC" deadline="2028-05-26 12:06:40.985133702 +0000 UTC"
Sep 11 16:16:50.517196 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:50.517097    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14947h49m50.468039017s"
Sep 11 16:16:50.600408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:50.599874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:50.600408 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.600026    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:16:50.600607 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:50.600509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:50.600660 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:50.600628    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:16:51.598320 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:51.597841    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:51.598320 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:51.598240    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:16:51.659858 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:51.659815    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal" event={"ID":"52aec3f96d0d8edc36cd253742d0de5e","Type":"ContainerStarted","Data":"aa22eecec3f9049f10d7e2f34acff9b7af2257d0994e98eeaf13e5cc647d65b8"}
Sep 11 16:16:51.671594 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:51.671559    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal" event={"ID":"83c5fb3c2cd6a94582ba9e97cbdde139","Type":"ContainerStarted","Data":"91f4ae2ce353700c367ca0fa792fe4d5d4dd502980effe9290ac2dadd0d7f558"}
Sep 11 16:16:51.708767 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:51.708703    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-197.ec2.internal" podStartSLOduration=3.708681158 podStartE2EDuration="3.708681158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:51.708128081 +0000 UTC m=+5.617929252" watchObservedRunningTime="2026-09-11 16:16:51.708681158 +0000 UTC m=+5.618482326"
Sep 11 16:16:52.106179 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:52.106137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:52.106370 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.106323    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:52.106441 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.106393    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs podName:e45bda4f-12ee-46a5-8d83-7a1602904679 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:56.10637252 +0000 UTC m=+10.016173671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs") pod "network-metrics-daemon-9sg48" (UID: "e45bda4f-12ee-46a5-8d83-7a1602904679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:52.208223 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:52.207573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:52.208223 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:52.207632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:52.208223 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.207778    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:52.208223 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.207854    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret podName:74c8f10b-e859-4c3e-b95a-5f58ee099241 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:56.207834059 +0000 UTC m=+10.117635206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret") pod "global-pull-secret-syncer-4gcmq" (UID: "74c8f10b-e859-4c3e-b95a-5f58ee099241") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:52.208223 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.207883    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:52.208223 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.207910    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:52.208223 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.207924    2576 projected.go:196] Error preparing data for projected volume kube-api-access-9prrq for pod openshift-network-diagnostics/network-check-target-dd2bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:52.208223 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.207980    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq podName:df08f2ff-be81-4cc3-966d-bc1be8960c4a nodeName:}" failed. No retries permitted until 2026-09-11 16:16:56.20796356 +0000 UTC m=+10.117764705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9prrq" (UniqueName: "kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq") pod "network-check-target-dd2bv" (UID: "df08f2ff-be81-4cc3-966d-bc1be8960c4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:52.601158 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:52.600552    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:52.601158 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.600653    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:16:52.601158 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:52.600708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:52.601158 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:52.600758    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:16:53.598429 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:53.598320    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:53.598585 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:53.598451    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:16:53.681048 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:53.681010    2576 generic.go:356] "Generic (PLEG): container finished" podID="52aec3f96d0d8edc36cd253742d0de5e" containerID="aa22eecec3f9049f10d7e2f34acff9b7af2257d0994e98eeaf13e5cc647d65b8" exitCode=0
Sep 11 16:16:53.681492 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:53.681071    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal" event={"ID":"52aec3f96d0d8edc36cd253742d0de5e","Type":"ContainerDied","Data":"aa22eecec3f9049f10d7e2f34acff9b7af2257d0994e98eeaf13e5cc647d65b8"}
Sep 11 16:16:54.598959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:54.597953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:54.598959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:54.597964    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:54.598959 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:54.598082    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:16:54.598959 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:54.598127    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:16:55.598558 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:55.598520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:55.598916 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:55.598658    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:16:56.141392 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:56.141354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:56.141554 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.141479    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:56.141554 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.141536    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs podName:e45bda4f-12ee-46a5-8d83-7a1602904679 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:04.141520725 +0000 UTC m=+18.051321893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs") pod "network-metrics-daemon-9sg48" (UID: "e45bda4f-12ee-46a5-8d83-7a1602904679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:56.242111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:56.242073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:56.242326 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:56.242123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:56.242326 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.242247    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:56.242326 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.242303    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret podName:74c8f10b-e859-4c3e-b95a-5f58ee099241 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:04.242282336 +0000 UTC m=+18.152083479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret") pod "global-pull-secret-syncer-4gcmq" (UID: "74c8f10b-e859-4c3e-b95a-5f58ee099241") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:56.242674 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.242653    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:56.242674 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.242677    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:56.242774 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.242690    2576 projected.go:196] Error preparing data for projected volume kube-api-access-9prrq for pod openshift-network-diagnostics/network-check-target-dd2bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:56.242774 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.242736    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq podName:df08f2ff-be81-4cc3-966d-bc1be8960c4a nodeName:}" failed. No retries permitted until 2026-09-11 16:17:04.242722562 +0000 UTC m=+18.152523707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9prrq" (UniqueName: "kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq") pod "network-check-target-dd2bv" (UID: "df08f2ff-be81-4cc3-966d-bc1be8960c4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:56.599576 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:56.599503    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:56.600010 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.599627    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:16:56.600010 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:56.599853    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:56.600010 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:56.599985    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:16:57.598170 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:57.598138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:57.598361 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:57.598274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:16:58.600580 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:58.600551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:16:58.601062 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:58.600675    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:16:58.601062 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:58.600736    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:16:58.601062 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:58.600814    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:16:59.598067 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:16:59.598045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:16:59.598169 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:16:59.598150    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:00.597620 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:00.597579    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:00.598081 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:00.597580    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:00.598081 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:00.597727    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:00.598081 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:00.597762    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:01.598389 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:01.598354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:01.598812 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:01.598457    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:02.598357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:02.598320    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:02.598546 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:02.598333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:02.598546 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:02.598447    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:02.598975 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:02.598540    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:03.598367 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:03.598327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:03.598567 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:03.598446    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:04.200214 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:04.200178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:04.200419 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.200353    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:04.200482 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.200425    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs podName:e45bda4f-12ee-46a5-8d83-7a1602904679 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:20.200409344 +0000 UTC m=+34.110210502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs") pod "network-metrics-daemon-9sg48" (UID: "e45bda4f-12ee-46a5-8d83-7a1602904679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:04.300996 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:04.300953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:04.300996 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:04.301005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:04.301254 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.301145    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:04.301254 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.301166    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:04.301254 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.301193    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:04.301254 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.301205    2576 projected.go:196] Error preparing data for projected volume kube-api-access-9prrq for pod openshift-network-diagnostics/network-check-target-dd2bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:04.301254 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.301215    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret podName:74c8f10b-e859-4c3e-b95a-5f58ee099241 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:20.301196205 +0000 UTC m=+34.210997346 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret") pod "global-pull-secret-syncer-4gcmq" (UID: "74c8f10b-e859-4c3e-b95a-5f58ee099241") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:04.301254 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.301253    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq podName:df08f2ff-be81-4cc3-966d-bc1be8960c4a nodeName:}" failed. No retries permitted until 2026-09-11 16:17:20.30123678 +0000 UTC m=+34.211037947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9prrq" (UniqueName: "kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq") pod "network-check-target-dd2bv" (UID: "df08f2ff-be81-4cc3-966d-bc1be8960c4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:04.597781 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:04.597711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:04.597947 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.597835    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:04.597947 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:04.597891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:04.598054 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:04.598000    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:05.597996 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:05.597962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:05.598441 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:05.598089    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:06.598350 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:06.598317    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:06.598735 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:06.598391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:06.598735 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:06.598417    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:06.598735 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:06.598513    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:07.598167 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.597575    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:07.598293 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:07.598207    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:07.709412 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.709385    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:17:07.710014 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.709675    2576 generic.go:356] "Generic (PLEG): container finished" podID="c911c447-3778-4e0c-bd78-297c361a8740" containerID="363127f51f316a62143d8be9b0376d173455b2242aee15d4f5e4868ed16b9f72" exitCode=1
Sep 11 16:17:07.710014 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.709731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerStarted","Data":"b4cd1a1e896a1f54ca6f27543fe1030ee95c09c53ff9243e183fc1c0094856c0"}
Sep 11 16:17:07.710014 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.709804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerStarted","Data":"0870dea354d660992e5a5156396869a6fa70f8e28c8b62e411ffd51b508ea121"}
Sep 11 16:17:07.710014 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.709855    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerStarted","Data":"4a49faf97bd981b50a1eeae687d491e9c30e843c781dd31b0e9d529fc7f88585"}
Sep 11 16:17:07.710014 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.709905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerDied","Data":"363127f51f316a62143d8be9b0376d173455b2242aee15d4f5e4868ed16b9f72"}
Sep 11 16:17:07.710014 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.709974    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerStarted","Data":"4b1598021686c2174f55139dfb15851218e18ace709dd73663ad9a5fc5056a01"}
Sep 11 16:17:07.711233 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.711213    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-48rfb" event={"ID":"37d25eeb-883f-4742-a291-61e8a28f44fe","Type":"ContainerStarted","Data":"89384f1a02d3297f04c175f6298ccca35597e38b195937e06ac7903bd70c27d5"}
Sep 11 16:17:07.712805 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.712781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal" event={"ID":"52aec3f96d0d8edc36cd253742d0de5e","Type":"ContainerStarted","Data":"f5c677d5e5989b2d6f602ae657a226d35344c89cd19818b64e793911b785a319"}
Sep 11 16:17:07.714207 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.714181    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mn7vw" event={"ID":"ea980b3b-40fd-4ea0-a4b3-70706ab8c942","Type":"ContainerStarted","Data":"e447b0cc6bb57d96f2c63425f2448e9da1d46819bb5b881cbf7110d1331a7aeb"}
Sep 11 16:17:07.715466 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.715447    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf" event={"ID":"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399","Type":"ContainerStarted","Data":"ba2281bfd30cc83889ca30be4fe075f23dc9410ee28c4d291ff36ffed3fd373c"}
Sep 11 16:17:07.716677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.716660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sjw2m" event={"ID":"5e2a6297-0738-43c3-9d88-7cfa8d9474d7","Type":"ContainerStarted","Data":"bea349c8ffa3e4e3c0863e8ab3246ffa6cbe700d2bdbf536c6fc6d31ce815996"}
Sep 11 16:17:07.717914 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.717896    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8bwsp" event={"ID":"db333929-74ef-449c-a79a-6878ef6051c1","Type":"ContainerStarted","Data":"5d12702599952e6b08d594adfea838a304a77d9f37830bb4a798bf513969d5ae"}
Sep 11 16:17:07.719320 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.719237    2576 generic.go:356] "Generic (PLEG): container finished" podID="a9648594-e0ca-46fd-ab95-86cfaa507bae" containerID="673a85789e4c9e06f578d64995d859b524c53f2583e25fba1a7047a7bdf33193" exitCode=0
Sep 11 16:17:07.719425 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.719325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7w7vl" event={"ID":"a9648594-e0ca-46fd-ab95-86cfaa507bae","Type":"ContainerDied","Data":"673a85789e4c9e06f578d64995d859b524c53f2583e25fba1a7047a7bdf33193"}
Sep 11 16:17:07.720852 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.720833    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jrpsp" event={"ID":"e725153a-43ec-44b9-a9bc-53ee55889dc9","Type":"ContainerStarted","Data":"d8c3dfea185b1e1c942933302cc4e0869f29a2f39b91f6bb0fb27922ccf38342"}
Sep 11 16:17:07.738203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.738162    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-48rfb" podStartSLOduration=6.576022081 podStartE2EDuration="20.738148481s" totalImagesPullingTime="14.1621264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:07.73779787 +0000 UTC m=+21.647599032" watchObservedRunningTime="2026-09-11 16:17:07.738148481 +0000 UTC m=+21.647949642"
Sep 11 16:17:07.781709 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.781654    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jrpsp" podStartSLOduration=2.629908301 podStartE2EDuration="20.781636177s" totalImagesPullingTime="18.151727876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:07.781531299 +0000 UTC m=+21.691332465" watchObservedRunningTime="2026-09-11 16:17:07.781636177 +0000 UTC m=+21.691437332"
Sep 11 16:17:07.782133 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.782096    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8bwsp" podStartSLOduration=2.595375237 podStartE2EDuration="20.78208216s" totalImagesPullingTime="18.186706923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:07.758757945 +0000 UTC m=+21.668559108" watchObservedRunningTime="2026-09-11 16:17:07.78208216 +0000 UTC m=+21.691883324"
Sep 11 16:17:07.831319 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.831265    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sjw2m" podStartSLOduration=2.678958657 podStartE2EDuration="20.831245973s" totalImagesPullingTime="18.152287316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:07.830347506 +0000 UTC m=+21.740148671" watchObservedRunningTime="2026-09-11 16:17:07.831245973 +0000 UTC m=+21.741047136"
Sep 11 16:17:07.858607 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.858563    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mn7vw" podStartSLOduration=2.638213306 podStartE2EDuration="20.858548814s" totalImagesPullingTime="18.220335508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:07.857464451 +0000 UTC m=+21.767265615" watchObservedRunningTime="2026-09-11 16:17:07.858548814 +0000 UTC m=+21.768349977"
Sep 11 16:17:07.884401 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.884353    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-197.ec2.internal" podStartSLOduration=19.884339683 podStartE2EDuration="19.884339683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:07.883897021 +0000 UTC m=+21.793698185" watchObservedRunningTime="2026-09-11 16:17:07.884339683 +0000 UTC m=+21.794140846"
Sep 11 16:17:07.903448 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.903423    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:17:07.904085 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:07.904066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:17:08.554179 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:08.554154    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 16:17:08.602085 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:08.602008    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:08.602228 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:08.602128    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:08.602228 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:08.602179    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:08.602302 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:08.602265    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:08.726141 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:08.726116    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:17:08.726574 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:08.726484    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerStarted","Data":"9ae5a43e098847e66361e8a251bd7efee1bce8c3b8026e22092ccfbd4cb39cca"}
Sep 11 16:17:08.728220 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:08.728199    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf" event={"ID":"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399","Type":"ContainerStarted","Data":"5db8416da9b72dc1aebfcf09484d3915273dd0ab205887f0188f342ae655b515"}
Sep 11 16:17:08.730045 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:08.730008    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zkl78" event={"ID":"51c1c801-a06c-4efb-95ec-75ad310b2565","Type":"ContainerStarted","Data":"79b66c05ceaeb7b6d8451781d9fc8389c3c6d9a0e62b37a076294dc08301d4f0"}
Sep 11 16:17:08.749825 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:08.749766    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zkl78" podStartSLOduration=3.544016911 podStartE2EDuration="21.749748241s" totalImagesPullingTime="18.20573133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:08.7488648 +0000 UTC m=+22.658665962" watchObservedRunningTime="2026-09-11 16:17:08.749748241 +0000 UTC m=+22.659549408"
Sep 11 16:17:09.214760 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:09.214722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:09.311646 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:09.311609    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:09.377364 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:09.377333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:09.513891 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:09.513768    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T16:17:08.554173414Z","UUID":"5e4de13e-2d88-43b6-b92c-7c1e678b80f1","Handler":null,"Name":"","Endpoint":""}
Sep 11 16:17:09.515537 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:09.515514    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 16:17:09.515634 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:09.515552    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 16:17:09.597949 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:09.597907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:09.598095 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:09.598043    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:09.735502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:09.735419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf" event={"ID":"8a7a6cd3-e0a9-4dd0-ad13-be32f0b2d399","Type":"ContainerStarted","Data":"ead4eaa34400154f57966e71e2f869bb258b36e7914205264bf523d8b4632cf8"}
Sep 11 16:17:09.735881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:09.735621    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 16:17:10.597951 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:10.597905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:10.598092 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:10.597908    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:10.598092 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:10.598040    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:10.598178 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:10.598146    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:10.741035 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:10.741007    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:17:10.741446 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:10.741398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerStarted","Data":"0cb5063ee71f1ed6ee002e3ab27cd03425ab3fa5025806b13ae7b81168d3e90a"}
Sep 11 16:17:11.597647 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:11.597618    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:11.597826 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:11.597714    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:12.598734 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.598452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:12.599360 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.598474    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:12.599360 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:12.598827    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:12.599360 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:12.598854    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:12.747669 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.747631    2576 generic.go:356] "Generic (PLEG): container finished" podID="a9648594-e0ca-46fd-ab95-86cfaa507bae" containerID="1f4125c6506d317057c00e9a947958fcbcfb9dcd5d53fdcfc408d104e732b1da" exitCode=0
Sep 11 16:17:12.747824 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.747715    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7w7vl" event={"ID":"a9648594-e0ca-46fd-ab95-86cfaa507bae","Type":"ContainerDied","Data":"1f4125c6506d317057c00e9a947958fcbcfb9dcd5d53fdcfc408d104e732b1da"}
Sep 11 16:17:12.751356 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.751338    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:17:12.751712 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.751682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerStarted","Data":"d0c56839c93439c83e2a1b3fe6326ac8efdf653cd4b690a6cb936bde856501d9"}
Sep 11 16:17:12.751972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.751952    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:17:12.752132 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.752117    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:17:12.752181 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.752131    2576 scope.go:117] "RemoveContainer" containerID="363127f51f316a62143d8be9b0376d173455b2242aee15d4f5e4868ed16b9f72"
Sep 11 16:17:12.768143 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.768123    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:17:12.768393 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.768373    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:17:12.772347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.772307    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzvqf" podStartSLOduration=5.62003549 podStartE2EDuration="25.772295478s" totalImagesPullingTime="20.152259988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:09.765178478 +0000 UTC m=+23.674979640" watchObservedRunningTime="2026-09-11 16:17:12.772295478 +0000 UTC m=+26.682096639"
Sep 11 16:17:12.819722 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:12.819693    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:17:13.597548 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:13.597515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:13.597748 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:13.597621    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:13.756197 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:13.756104    2576 generic.go:356] "Generic (PLEG): container finished" podID="a9648594-e0ca-46fd-ab95-86cfaa507bae" containerID="fcb98526da29c07dbecd7fd9b307900807da802c71b947f1ac0c9d28af122aea" exitCode=0
Sep 11 16:17:13.756758 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:13.756189    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7w7vl" event={"ID":"a9648594-e0ca-46fd-ab95-86cfaa507bae","Type":"ContainerDied","Data":"fcb98526da29c07dbecd7fd9b307900807da802c71b947f1ac0c9d28af122aea"}
Sep 11 16:17:13.760163 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:13.760147    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:17:13.760557 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:13.760539    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" event={"ID":"c911c447-3778-4e0c-bd78-297c361a8740","Type":"ContainerStarted","Data":"e1bbe9cb4a3287e762ea81f77b1fcac3ef8cc38473c312e0fca506f5e6edfad6"}
Sep 11 16:17:13.811371 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:13.811322    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5" podStartSLOduration=8.568066374 podStartE2EDuration="26.811305059s" totalImagesPullingTime="18.243238685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:13.809637526 +0000 UTC m=+27.719438689" watchObservedRunningTime="2026-09-11 16:17:13.811305059 +0000 UTC m=+27.721106220"
Sep 11 16:17:14.342071 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:14.341678    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4gcmq"]
Sep 11 16:17:14.342303 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:14.342102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:14.342303 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:14.342203    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:14.342560 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:14.342541    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9sg48"]
Sep 11 16:17:14.342648 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:14.342626    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:14.342702 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:14.342690    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:14.343142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:14.343121    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dd2bv"]
Sep 11 16:17:14.343246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:14.343212    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:14.343323 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:14.343303    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:14.688043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:14.687957    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:14.765059 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:14.765031    2576 generic.go:356] "Generic (PLEG): container finished" podID="a9648594-e0ca-46fd-ab95-86cfaa507bae" containerID="1d6fa19e12e01bafbb2860986f5dcafc1a71a8054a8803c8b642bb2c0e887a80" exitCode=0
Sep 11 16:17:14.765412 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:14.765119    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7w7vl" event={"ID":"a9648594-e0ca-46fd-ab95-86cfaa507bae","Type":"ContainerDied","Data":"1d6fa19e12e01bafbb2860986f5dcafc1a71a8054a8803c8b642bb2c0e887a80"}
Sep 11 16:17:15.598580 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:15.598545    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:15.598759 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:15.598653    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:15.598759 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:15.598708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:15.598881 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:15.598827    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:16.243099 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:16.242966    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:16.602644 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:16.602615    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:16.602762 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:16.602699    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:17.598071 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:17.598040    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:17.598527 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:17.598039    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:17.598527 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:17.598157    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:17.598527 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:17.598263    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:18.590554 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:18.590518    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:17:18.590826 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:18.590719    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 16:17:18.591305 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:18.591284    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-48rfb"
Sep 11 16:17:18.599602 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:18.599326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:18.599602 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:18.599455    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9sg48" podUID="e45bda4f-12ee-46a5-8d83-7a1602904679"
Sep 11 16:17:18.676820 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:18.676787    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:19.598633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:19.598420    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:19.598776 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:19.598419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:19.598776 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:19.598656    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dd2bv" podUID="df08f2ff-be81-4cc3-966d-bc1be8960c4a"
Sep 11 16:17:19.598776 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:19.598756    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4gcmq" podUID="74c8f10b-e859-4c3e-b95a-5f58ee099241"
Sep 11 16:17:19.780459 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:19.780427    2576 generic.go:356] "Generic (PLEG): container finished" podID="a9648594-e0ca-46fd-ab95-86cfaa507bae" containerID="c01c0efcc332c9d02cdf6b96e34cfa48bee967e737aee979e26907622c1bb97e" exitCode=0
Sep 11 16:17:19.780867 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:19.780494    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7w7vl" event={"ID":"a9648594-e0ca-46fd-ab95-86cfaa507bae","Type":"ContainerDied","Data":"c01c0efcc332c9d02cdf6b96e34cfa48bee967e737aee979e26907622c1bb97e"}
Sep 11 16:17:20.214632 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.214607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:20.214769 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.214758    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:20.214828 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.214818    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs podName:e45bda4f-12ee-46a5-8d83-7a1602904679 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:52.214801455 +0000 UTC m=+66.124602601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs") pod "network-metrics-daemon-9sg48" (UID: "e45bda4f-12ee-46a5-8d83-7a1602904679") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:20.315142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.315107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:20.315269 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.315168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:20.315269 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.315245    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:20.315339 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.315270    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:20.315339 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.315288    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:20.315339 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.315300    2576 projected.go:196] Error preparing data for projected volume kube-api-access-9prrq for pod openshift-network-diagnostics/network-check-target-dd2bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:20.315339 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.315301    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret podName:74c8f10b-e859-4c3e-b95a-5f58ee099241 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:52.315287171 +0000 UTC m=+66.225088311 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret") pod "global-pull-secret-syncer-4gcmq" (UID: "74c8f10b-e859-4c3e-b95a-5f58ee099241") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:20.315339 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.315337    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq podName:df08f2ff-be81-4cc3-966d-bc1be8960c4a nodeName:}" failed. No retries permitted until 2026-09-11 16:17:52.315326834 +0000 UTC m=+66.225127974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9prrq" (UniqueName: "kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq") pod "network-check-target-dd2bv" (UID: "df08f2ff-be81-4cc3-966d-bc1be8960c4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:20.363249 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.363217    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-197.ec2.internal" event="NodeReady"
Sep 11 16:17:20.363374 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.363322    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 16:17:20.423418 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.423390    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rzth4"]
Sep 11 16:17:20.426399 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.426384    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dttkg"]
Sep 11 16:17:20.426516 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.426501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.429153 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.429133    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6q2x\""
Sep 11 16:17:20.429372 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.429349    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 16:17:20.429452 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.429395    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dttkg"
Sep 11 16:17:20.429633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.429614    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 16:17:20.432152 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.432137    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qzvwg\""
Sep 11 16:17:20.432336 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.432321    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 16:17:20.432621 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.432609    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 16:17:20.432669 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.432646    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 16:17:20.436203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.436184    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dttkg"]
Sep 11 16:17:20.439901 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.439838    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rzth4"]
Sep 11 16:17:20.519686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.518087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.519686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.518194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b250f482-a270-4c37-89a0-69fee0abc0ae-config-volume\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.519686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.518244    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b250f482-a270-4c37-89a0-69fee0abc0ae-tmp-dir\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.519686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.518283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w25sj\" (UniqueName: \"kubernetes.io/projected/b250f482-a270-4c37-89a0-69fee0abc0ae-kube-api-access-w25sj\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.519686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.518325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e-cert\") pod \"ingress-canary-dttkg\" (UID: \"52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e\") " pod="openshift-ingress-canary/ingress-canary-dttkg"
Sep 11 16:17:20.519686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.518353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvps\" (UniqueName: \"kubernetes.io/projected/52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e-kube-api-access-dsvps\") pod \"ingress-canary-dttkg\" (UID: \"52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e\") " pod="openshift-ingress-canary/ingress-canary-dttkg"
Sep 11 16:17:20.597619 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.597588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:20.604346 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.604326    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:17:20.604901 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.604883    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kx52\""
Sep 11 16:17:20.618735 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.618714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.618817 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.618760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b250f482-a270-4c37-89a0-69fee0abc0ae-config-volume\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.618817 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.618788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b250f482-a270-4c37-89a0-69fee0abc0ae-tmp-dir\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.618892 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.618856    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:20.618930 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:20.618904    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls podName:b250f482-a270-4c37-89a0-69fee0abc0ae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:21.118889994 +0000 UTC m=+35.028691133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls") pod "dns-default-rzth4" (UID: "b250f482-a270-4c37-89a0-69fee0abc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:20.619025 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.618978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w25sj\" (UniqueName: \"kubernetes.io/projected/b250f482-a270-4c37-89a0-69fee0abc0ae-kube-api-access-w25sj\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.619079 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.619039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e-cert\") pod \"ingress-canary-dttkg\" (UID: \"52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e\") " pod="openshift-ingress-canary/ingress-canary-dttkg"
Sep 11 16:17:20.619079 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.619071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvps\" (UniqueName: \"kubernetes.io/projected/52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e-kube-api-access-dsvps\") pod \"ingress-canary-dttkg\" (UID: \"52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e\") " pod="openshift-ingress-canary/ingress-canary-dttkg"
Sep 11 16:17:20.619175 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.619150    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b250f482-a270-4c37-89a0-69fee0abc0ae-tmp-dir\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.619409 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.619387    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b250f482-a270-4c37-89a0-69fee0abc0ae-config-volume\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.623224 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.623203    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e-cert\") pod \"ingress-canary-dttkg\" (UID: \"52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e\") " pod="openshift-ingress-canary/ingress-canary-dttkg"
Sep 11 16:17:20.648767 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.648738    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvps\" (UniqueName: \"kubernetes.io/projected/52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e-kube-api-access-dsvps\") pod \"ingress-canary-dttkg\" (UID: \"52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e\") " pod="openshift-ingress-canary/ingress-canary-dttkg"
Sep 11 16:17:20.649387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.649367    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:20.651951 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.651908    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w25sj\" (UniqueName: \"kubernetes.io/projected/b250f482-a270-4c37-89a0-69fee0abc0ae-kube-api-access-w25sj\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:20.742687 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.742653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dttkg"
Sep 11 16:17:20.744039 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.744008    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:20.787428 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.787345    2576 generic.go:356] "Generic (PLEG): container finished" podID="a9648594-e0ca-46fd-ab95-86cfaa507bae" containerID="4aad655cda3a0a208b40a614bd82e2d1697e6b434af1a9697a52112199dda4a7" exitCode=0
Sep 11 16:17:20.787428 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:20.787404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7w7vl" event={"ID":"a9648594-e0ca-46fd-ab95-86cfaa507bae","Type":"ContainerDied","Data":"4aad655cda3a0a208b40a614bd82e2d1697e6b434af1a9697a52112199dda4a7"}
Sep 11 16:17:21.027544 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.027364    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dttkg"]
Sep 11 16:17:21.030385 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:17:21.030358    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a2aecb_8c51_4bbc_80ff_7ccc0d0c769e.slice/crio-3248bf56000e3b6376dad0493b3247ec61f7a4d5b591fa06e03f9a3a8c90883b WatchSource:0}: Error finding container 3248bf56000e3b6376dad0493b3247ec61f7a4d5b591fa06e03f9a3a8c90883b: Status 404 returned error can't find the container with id 3248bf56000e3b6376dad0493b3247ec61f7a4d5b591fa06e03f9a3a8c90883b
Sep 11 16:17:21.124226 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.124104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:21.124351 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:21.124249    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:21.124351 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:21.124312    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls podName:b250f482-a270-4c37-89a0-69fee0abc0ae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:22.124295479 +0000 UTC m=+36.034096619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls") pod "dns-default-rzth4" (UID: "b250f482-a270-4c37-89a0-69fee0abc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:21.598052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.598019    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:21.598216 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.598019    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:21.602037 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.602009    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xprtk\""
Sep 11 16:17:21.602037 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.602033    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:17:21.602223 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.602043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:17:21.602223 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.602078    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:17:21.794694 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.794657    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7w7vl" event={"ID":"a9648594-e0ca-46fd-ab95-86cfaa507bae","Type":"ContainerStarted","Data":"452ce86b8b5cd5b233f20fd8acbadacd15d37bc1af5331fa5dec2ed74d0ab31b"}
Sep 11 16:17:21.795863 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.795841    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dttkg" event={"ID":"52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e","Type":"ContainerStarted","Data":"3248bf56000e3b6376dad0493b3247ec61f7a4d5b591fa06e03f9a3a8c90883b"}
Sep 11 16:17:21.829991 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:21.829921    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7w7vl" podStartSLOduration=10.450286132 podStartE2EDuration="34.829904342s" totalImagesPullingTime="24.37961821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:21.829506553 +0000 UTC m=+35.739307716" watchObservedRunningTime="2026-09-11 16:17:21.829904342 +0000 UTC m=+35.739705515"
Sep 11 16:17:22.015282 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:22.015248    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:22.106254 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:22.106223    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:22.130813 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:22.130775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:22.131036 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:22.130983    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:22.131160 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:22.131082    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls podName:b250f482-a270-4c37-89a0-69fee0abc0ae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:24.13104962 +0000 UTC m=+38.040850774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls") pod "dns-default-rzth4" (UID: "b250f482-a270-4c37-89a0-69fee0abc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:22.208305 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:22.208267    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:22.311378 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:22.311282    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:22.412966 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:22.412909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:22.996403 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:22.996370    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:23.340047 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:23.339958    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:23.409464 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:23.409428    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:24.145200 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:24.145105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:24.145675 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:24.145286    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:24.145675 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:24.145375    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls podName:b250f482-a270-4c37-89a0-69fee0abc0ae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:28.145350157 +0000 UTC m=+42.055151303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls") pod "dns-default-rzth4" (UID: "b250f482-a270-4c37-89a0-69fee0abc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:24.803568 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:24.803531    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dttkg" event={"ID":"52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e","Type":"ContainerStarted","Data":"5f4ffb4acf8a674c23448b28b3e79326b880787829da498b2a424180f4d4298f"}
Sep 11 16:17:24.826106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:24.826056    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dttkg" podStartSLOduration=1.970075724 podStartE2EDuration="4.826042681s" totalImagesPullingTime="2.855966957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:24.824751584 +0000 UTC m=+38.734552747" watchObservedRunningTime="2026-09-11 16:17:24.826042681 +0000 UTC m=+38.735843845"
Sep 11 16:17:24.952787 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:24.952753    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:25.173803 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:25.173721    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:25.295485 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:25.295456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:28.174717 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:28.174637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:28.175081 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:28.174782    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:28.175081 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:28.174843    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls podName:b250f482-a270-4c37-89a0-69fee0abc0ae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:36.174826883 +0000 UTC m=+50.084628026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls") pod "dns-default-rzth4" (UID: "b250f482-a270-4c37-89a0-69fee0abc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:34.158478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:34.158444    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:34.264001 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:34.263964    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:35.645271 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:35.645232    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:35.725635 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:35.725605    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:36.230773 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:36.230734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:36.230994 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:36.230925    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:36.231077 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:36.231065    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls podName:b250f482-a270-4c37-89a0-69fee0abc0ae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:52.231041852 +0000 UTC m=+66.140843006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls") pod "dns-default-rzth4" (UID: "b250f482-a270-4c37-89a0-69fee0abc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:37.074479 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:37.074442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:37.173476 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:37.173441    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:39.464496 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:39.464468    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:39.562280 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:39.562243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:44.776246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:44.776214    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkd5"
Sep 11 16:17:44.958880 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:44.958841    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:45.043777 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:45.043689    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:51.263245 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.263207    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"]
Sep 11 16:17:51.294621 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.294588    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-6jh9m"]
Sep 11 16:17:51.294762 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.294734    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:51.297657 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.297631    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 16:17:51.297800 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.297631    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 16:17:51.298041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.298025    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pvmtg\""
Sep 11 16:17:51.298103 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.298077    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:17:51.308517 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.308496    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"]
Sep 11 16:17:51.308615 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.308563    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.311550 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.311533    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 11 16:17:51.311670 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.311601    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9lgnd\""
Sep 11 16:17:51.311988 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.311974    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 11 16:17:51.312455 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.312438    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:17:51.312888 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.312872    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 11 16:17:51.326562 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.326542    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 11 16:17:51.330628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.330612    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"]
Sep 11 16:17:51.330723 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.330650    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"]
Sep 11 16:17:51.330723 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.330674    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-6jh9m"]
Sep 11 16:17:51.330723 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.330715    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.333878 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.333859    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 16:17:51.334055 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.334036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 16:17:51.334134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.334043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qj6tp\""
Sep 11 16:17:51.334423 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.334406    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:17:51.334500 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.334437    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 16:17:51.338826 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.338809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ggz\" (UniqueName: \"kubernetes.io/projected/9065a2bc-304a-44a4-ab32-fe1cd9e41a59-kube-api-access-f9ggz\") pod \"service-ca-operator-cf7c776b7-htl5r\" (UID: \"9065a2bc-304a-44a4-ab32-fe1cd9e41a59\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.338907 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.338850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f563067-de11-4c7f-92eb-0cdefa727b88-serving-cert\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.338907 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.338865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f563067-de11-4c7f-92eb-0cdefa727b88-trusted-ca\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.339003 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.338914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9065a2bc-304a-44a4-ab32-fe1cd9e41a59-serving-cert\") pod \"service-ca-operator-cf7c776b7-htl5r\" (UID: \"9065a2bc-304a-44a4-ab32-fe1cd9e41a59\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.339003 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.338977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9065a2bc-304a-44a4-ab32-fe1cd9e41a59-config\") pod \"service-ca-operator-cf7c776b7-htl5r\" (UID: \"9065a2bc-304a-44a4-ab32-fe1cd9e41a59\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.339003 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.338997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:51.339187 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.339028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvb8b\" (UniqueName: \"kubernetes.io/projected/0f563067-de11-4c7f-92eb-0cdefa727b88-kube-api-access-pvb8b\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.339187 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.339057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4wl5\" (UniqueName: \"kubernetes.io/projected/a7cbce59-f327-472b-a883-f88c2173e231-kube-api-access-b4wl5\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:51.339187 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.339074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0f563067-de11-4c7f-92eb-0cdefa727b88-config\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.439625 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.439594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wl5\" (UniqueName: \"kubernetes.io/projected/a7cbce59-f327-472b-a883-f88c2173e231-kube-api-access-b4wl5\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:51.439625 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.439626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0f563067-de11-4c7f-92eb-0cdefa727b88-config\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.439834 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.439662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ggz\" (UniqueName: \"kubernetes.io/projected/9065a2bc-304a-44a4-ab32-fe1cd9e41a59-kube-api-access-f9ggz\") pod \"service-ca-operator-cf7c776b7-htl5r\" (UID: \"9065a2bc-304a-44a4-ab32-fe1cd9e41a59\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.439834 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.439715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f563067-de11-4c7f-92eb-0cdefa727b88-serving-cert\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.439834 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.439739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f563067-de11-4c7f-92eb-0cdefa727b88-trusted-ca\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.439834 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.439771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9065a2bc-304a-44a4-ab32-fe1cd9e41a59-serving-cert\") pod \"service-ca-operator-cf7c776b7-htl5r\" (UID: \"9065a2bc-304a-44a4-ab32-fe1cd9e41a59\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.439834 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.439819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9065a2bc-304a-44a4-ab32-fe1cd9e41a59-config\") pod \"service-ca-operator-cf7c776b7-htl5r\" (UID: \"9065a2bc-304a-44a4-ab32-fe1cd9e41a59\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.440133 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.439842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:51.440133 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.439883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvb8b\" (UniqueName: \"kubernetes.io/projected/0f563067-de11-4c7f-92eb-0cdefa727b88-kube-api-access-pvb8b\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.440133 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:51.440081    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:17:51.440282 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:51.440151    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls podName:a7cbce59-f327-472b-a883-f88c2173e231 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:51.940133207 +0000 UTC m=+65.849934354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tmfk2" (UID: "a7cbce59-f327-472b-a883-f88c2173e231") : secret "samples-operator-tls" not found
Sep 11 16:17:51.440461 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.440437    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0f563067-de11-4c7f-92eb-0cdefa727b88-config\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.440641 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.440618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f563067-de11-4c7f-92eb-0cdefa727b88-trusted-ca\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.443331 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.443310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f563067-de11-4c7f-92eb-0cdefa727b88-serving-cert\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.452539 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.452519    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9065a2bc-304a-44a4-ab32-fe1cd9e41a59-serving-cert\") pod \"service-ca-operator-cf7c776b7-htl5r\" (UID: \"9065a2bc-304a-44a4-ab32-fe1cd9e41a59\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.452679 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.452662    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9065a2bc-304a-44a4-ab32-fe1cd9e41a59-config\") pod \"service-ca-operator-cf7c776b7-htl5r\" (UID: \"9065a2bc-304a-44a4-ab32-fe1cd9e41a59\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.467514 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.467494    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wl5\" (UniqueName: \"kubernetes.io/projected/a7cbce59-f327-472b-a883-f88c2173e231-kube-api-access-b4wl5\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:51.468268 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.468249    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvb8b\" (UniqueName: \"kubernetes.io/projected/0f563067-de11-4c7f-92eb-0cdefa727b88-kube-api-access-pvb8b\") pod \"console-operator-696884fb9c-6jh9m\" (UID: \"0f563067-de11-4c7f-92eb-0cdefa727b88\") " pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.480997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.480980    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ggz\" (UniqueName: \"kubernetes.io/projected/9065a2bc-304a-44a4-ab32-fe1cd9e41a59-kube-api-access-f9ggz\") pod \"service-ca-operator-cf7c776b7-htl5r\" (UID: \"9065a2bc-304a-44a4-ab32-fe1cd9e41a59\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.618796 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.618709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:51.638715 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.638693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"
Sep 11 16:17:51.701329 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.701231    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:51.806440 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.806406    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:51.904665 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.904589    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-6jh9m"]
Sep 11 16:17:51.907827 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:17:51.907797    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f563067_de11_4c7f_92eb_0cdefa727b88.slice/crio-fed52a5ba539dbc39e86fd1976d5139dff0b3e60109f41b5dfae6cd051d2a5ec WatchSource:0}: Error finding container fed52a5ba539dbc39e86fd1976d5139dff0b3e60109f41b5dfae6cd051d2a5ec: Status 404 returned error can't find the container with id fed52a5ba539dbc39e86fd1976d5139dff0b3e60109f41b5dfae6cd051d2a5ec
Sep 11 16:17:51.943269 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.943243    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r"]
Sep 11 16:17:51.944499 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:51.944439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:51.944557 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:51.944543    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:17:51.944641 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:51.944597    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls podName:a7cbce59-f327-472b-a883-f88c2173e231 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:52.944583785 +0000 UTC m=+66.854384925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tmfk2" (UID: "a7cbce59-f327-472b-a883-f88c2173e231") : secret "samples-operator-tls" not found
Sep 11 16:17:51.946456 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:17:51.946431    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9065a2bc_304a_44a4_ab32_fe1cd9e41a59.slice/crio-4ab2a25748829ade0a71be60e07ca377d56e64c78454c38f4993ee9b20f335fa WatchSource:0}: Error finding container 4ab2a25748829ade0a71be60e07ca377d56e64c78454c38f4993ee9b20f335fa: Status 404 returned error can't find the container with id 4ab2a25748829ade0a71be60e07ca377d56e64c78454c38f4993ee9b20f335fa
Sep 11 16:17:52.246464 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.246432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:17:52.246646 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.246478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:17:52.246646 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:52.246584    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:52.246646 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:52.246629    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls podName:b250f482-a270-4c37-89a0-69fee0abc0ae nodeName:}" failed. No retries permitted until 2026-09-11 16:18:24.246616516 +0000 UTC m=+98.156417656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls") pod "dns-default-rzth4" (UID: "b250f482-a270-4c37-89a0-69fee0abc0ae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:52.249553 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.249536    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:17:52.256993 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:52.256980    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 16:17:52.257051 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:52.257032    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs podName:e45bda4f-12ee-46a5-8d83-7a1602904679 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:56.257015114 +0000 UTC m=+130.166816255 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs") pod "network-metrics-daemon-9sg48" (UID: "e45bda4f-12ee-46a5-8d83-7a1602904679") : secret "metrics-daemon-secret" not found
Sep 11 16:17:52.346985 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.346931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:52.347400 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.346995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:52.350037 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.350018    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:17:52.350086 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.350039    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:17:52.359988 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.359967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74c8f10b-e859-4c3e-b95a-5f58ee099241-original-pull-secret\") pod \"global-pull-secret-syncer-4gcmq\" (UID: \"74c8f10b-e859-4c3e-b95a-5f58ee099241\") " pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:52.360550 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.360535    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:17:52.370307 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.370286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prrq\" (UniqueName: \"kubernetes.io/projected/df08f2ff-be81-4cc3-966d-bc1be8960c4a-kube-api-access-9prrq\") pod \"network-check-target-dd2bv\" (UID: \"df08f2ff-be81-4cc3-966d-bc1be8960c4a\") " pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:52.508271 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.508195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4gcmq"
Sep 11 16:17:52.515126 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.515098    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xprtk\""
Sep 11 16:17:52.522587 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.522379    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:52.802428 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.801559    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4gcmq"]
Sep 11 16:17:52.805295 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:17:52.805267    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c8f10b_e859_4c3e_b95a_5f58ee099241.slice/crio-687395b16690c345b68b1bb6969f06497ab2824a2ccc0a04c53b2f7cce43ad90 WatchSource:0}: Error finding container 687395b16690c345b68b1bb6969f06497ab2824a2ccc0a04c53b2f7cce43ad90: Status 404 returned error can't find the container with id 687395b16690c345b68b1bb6969f06497ab2824a2ccc0a04c53b2f7cce43ad90
Sep 11 16:17:52.814917 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.814893    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dd2bv"]
Sep 11 16:17:52.818366 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:17:52.818335    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf08f2ff_be81_4cc3_966d_bc1be8960c4a.slice/crio-a9f938e9bc7e518e65158085e825ac29aa2d9620c683cfa5159b67e19167d0cc WatchSource:0}: Error finding container a9f938e9bc7e518e65158085e825ac29aa2d9620c683cfa5159b67e19167d0cc: Status 404 returned error can't find the container with id a9f938e9bc7e518e65158085e825ac29aa2d9620c683cfa5159b67e19167d0cc
Sep 11 16:17:52.870647 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.870610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dd2bv" event={"ID":"df08f2ff-be81-4cc3-966d-bc1be8960c4a","Type":"ContainerStarted","Data":"a9f938e9bc7e518e65158085e825ac29aa2d9620c683cfa5159b67e19167d0cc"}
Sep 11 16:17:52.872392 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.872364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4gcmq" event={"ID":"74c8f10b-e859-4c3e-b95a-5f58ee099241","Type":"ContainerStarted","Data":"687395b16690c345b68b1bb6969f06497ab2824a2ccc0a04c53b2f7cce43ad90"}
Sep 11 16:17:52.873971 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.873924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" event={"ID":"0f563067-de11-4c7f-92eb-0cdefa727b88","Type":"ContainerStarted","Data":"fed52a5ba539dbc39e86fd1976d5139dff0b3e60109f41b5dfae6cd051d2a5ec"}
Sep 11 16:17:52.875536 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.875509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r" event={"ID":"9065a2bc-304a-44a4-ab32-fe1cd9e41a59","Type":"ContainerStarted","Data":"4ab2a25748829ade0a71be60e07ca377d56e64c78454c38f4993ee9b20f335fa"}
Sep 11 16:17:52.950627 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:52.950593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:52.950804 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:52.950785    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:17:52.950865 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:52.950856    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls podName:a7cbce59-f327-472b-a883-f88c2173e231 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:54.950835763 +0000 UTC m=+68.860636908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tmfk2" (UID: "a7cbce59-f327-472b-a883-f88c2173e231") : secret "samples-operator-tls" not found
Sep 11 16:17:54.077548 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:54.077468    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:54.968204 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:54.968119    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:54.968339 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:54.968276    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:17:54.968399 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:54.968350    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls podName:a7cbce59-f327-472b-a883-f88c2173e231 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:58.968327931 +0000 UTC m=+72.878129078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tmfk2" (UID: "a7cbce59-f327-472b-a883-f88c2173e231") : secret "samples-operator-tls" not found
Sep 11 16:17:55.007448 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.007418    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:55.085502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.085422    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:55.353793 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.353707    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:55.374957 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.374912    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bbldx"]
Sep 11 16:17:55.379958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.379740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.383228 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.383049    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 16:17:55.384379 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.384106    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t8wqd\""
Sep 11 16:17:55.384379 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.384117    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 16:17:55.384379 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.384164    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 16:17:55.384379 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.384117    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 16:17:55.394155 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.393039    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bbldx"]
Sep 11 16:17:55.431033 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.430958    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:55.472205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.472173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a163692-fd52-4032-8ae6-3966412e2561-data-volume\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.472205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.472205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.472444 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.472234    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.472444 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.472311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a163692-fd52-4032-8ae6-3966412e2561-crio-socket\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.472444 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.472348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cfqk\" (UniqueName: \"kubernetes.io/projected/1a163692-fd52-4032-8ae6-3966412e2561-kube-api-access-9cfqk\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.573112 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.573060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cfqk\" (UniqueName: \"kubernetes.io/projected/1a163692-fd52-4032-8ae6-3966412e2561-kube-api-access-9cfqk\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.573298 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.573150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a163692-fd52-4032-8ae6-3966412e2561-data-volume\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.573298 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.573177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.573298 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.573204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.573298 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.573254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a163692-fd52-4032-8ae6-3966412e2561-crio-socket\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.573458 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.573355    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a163692-fd52-4032-8ae6-3966412e2561-crio-socket\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.573725 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:55.573580    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:17:55.573725 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:55.573614    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:55.573725 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:55.573667    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:56.073645033 +0000 UTC m=+69.983446197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : configmap "kube-rbac-proxy" not found
Sep 11 16:17:55.573725 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:55.573687    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:56.073678158 +0000 UTC m=+69.983479309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:55.574028 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.573878    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a163692-fd52-4032-8ae6-3966412e2561-data-volume\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.586657 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.586625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cfqk\" (UniqueName: \"kubernetes.io/projected/1a163692-fd52-4032-8ae6-3966412e2561-kube-api-access-9cfqk\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:55.892014 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.891977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r" event={"ID":"9065a2bc-304a-44a4-ab32-fe1cd9e41a59","Type":"ContainerStarted","Data":"2ec32b585d83a9dd160a6e31e099d69e3917254c01b6c46b1c2677a25048a667"}
Sep 11 16:17:55.894153 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.894130    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/0.log"
Sep 11 16:17:55.894275 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.894171    2576 generic.go:356] "Generic (PLEG): container finished" podID="0f563067-de11-4c7f-92eb-0cdefa727b88" containerID="90077fcb6b72b4f306a1d1c2481605693c6136e1fd78c6b46352d885aa773ee3" exitCode=255
Sep 11 16:17:55.894275 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.894202    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" event={"ID":"0f563067-de11-4c7f-92eb-0cdefa727b88","Type":"ContainerDied","Data":"90077fcb6b72b4f306a1d1c2481605693c6136e1fd78c6b46352d885aa773ee3"}
Sep 11 16:17:55.894511 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.894495    2576 scope.go:117] "RemoveContainer" containerID="90077fcb6b72b4f306a1d1c2481605693c6136e1fd78c6b46352d885aa773ee3"
Sep 11 16:17:55.913105 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.913054    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r" podStartSLOduration=2.051658934 podStartE2EDuration="4.91303671s" totalImagesPullingTime="2.861377776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:55.912639249 +0000 UTC m=+69.822440414" watchObservedRunningTime="2026-09-11 16:17:55.91303671 +0000 UTC m=+69.822838055"
Sep 11 16:17:55.998911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:55.998881    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv"]
Sep 11 16:17:56.002498 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.002474    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv"
Sep 11 16:17:56.005409 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.005388    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 16:17:56.005551 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.005486    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v84tz\""
Sep 11 16:17:56.005768 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.005749    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 16:17:56.014063 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.014020    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv"]
Sep 11 16:17:56.053605 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.053575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:56.077470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.077443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5kt\" (UniqueName: \"kubernetes.io/projected/6eb31607-be58-459e-b1bf-a1b96c2e68d4-kube-api-access-4z5kt\") pod \"migrator-8b4d697c5-wdxwv\" (UID: \"6eb31607-be58-459e-b1bf-a1b96c2e68d4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv"
Sep 11 16:17:56.077637 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.077485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:56.077637 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.077516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:56.077637 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:56.077608    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:56.077637 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:56.077612    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:17:56.077833 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:56.077659    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:57.077643956 +0000 UTC m=+70.987445096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:56.077833 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:56.077676    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:57.077666648 +0000 UTC m=+70.987467787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : configmap "kube-rbac-proxy" not found
Sep 11 16:17:56.159014 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.158907    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:56.179009 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.178976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5kt\" (UniqueName: \"kubernetes.io/projected/6eb31607-be58-459e-b1bf-a1b96c2e68d4-kube-api-access-4z5kt\") pod \"migrator-8b4d697c5-wdxwv\" (UID: \"6eb31607-be58-459e-b1bf-a1b96c2e68d4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv"
Sep 11 16:17:56.191422 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.191389    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5kt\" (UniqueName: \"kubernetes.io/projected/6eb31607-be58-459e-b1bf-a1b96c2e68d4-kube-api-access-4z5kt\") pod \"migrator-8b4d697c5-wdxwv\" (UID: \"6eb31607-be58-459e-b1bf-a1b96c2e68d4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv"
Sep 11 16:17:56.314307 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.314271    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv"
Sep 11 16:17:56.659205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.659169    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz"]
Sep 11 16:17:56.659397 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.659229    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:56.662430 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.662408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz"
Sep 11 16:17:56.665161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.665130    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dzqqp\""
Sep 11 16:17:56.675052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.675019    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz"]
Sep 11 16:17:56.683443 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.683414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drfn6\" (UniqueName: \"kubernetes.io/projected/61eeb2f0-b6c0-4286-80b8-3fc3268a8d90-kube-api-access-drfn6\") pod \"network-check-source-6f8b768455-hwvmz\" (UID: \"61eeb2f0-b6c0-4286-80b8-3fc3268a8d90\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz"
Sep 11 16:17:56.738037 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.738002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:56.784658 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.784618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drfn6\" (UniqueName: \"kubernetes.io/projected/61eeb2f0-b6c0-4286-80b8-3fc3268a8d90-kube-api-access-drfn6\") pod \"network-check-source-6f8b768455-hwvmz\" (UID: \"61eeb2f0-b6c0-4286-80b8-3fc3268a8d90\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz"
Sep 11 16:17:56.798878 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.798852    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drfn6\" (UniqueName: \"kubernetes.io/projected/61eeb2f0-b6c0-4286-80b8-3fc3268a8d90-kube-api-access-drfn6\") pod \"network-check-source-6f8b768455-hwvmz\" (UID: \"61eeb2f0-b6c0-4286-80b8-3fc3268a8d90\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz"
Sep 11 16:17:56.973383 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:56.973339    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz"
Sep 11 16:17:57.088049 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.088012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:57.088049 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.088055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:57.088299 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:57.088158    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:17:57.088299 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:57.088179    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:57.088299 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:57.088238    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:59.088215872 +0000 UTC m=+72.998017022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : configmap "kube-rbac-proxy" not found
Sep 11 16:17:57.088299 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:57.088259    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:59.088249056 +0000 UTC m=+72.998050197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:57.224151 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.224065    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8bwsp_db333929-74ef-449c-a79a-6878ef6051c1/dns-node-resolver/0.log"
Sep 11 16:17:57.355845 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.355807    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:57.460955 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.460900    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:57.907803 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.907779    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/0.log"
Sep 11 16:17:57.907903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.907867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" event={"ID":"0f563067-de11-4c7f-92eb-0cdefa727b88","Type":"ContainerStarted","Data":"efd5d1db6a47091a0155d965e5c8d2880bd16dcba21af1e897691638b097f1c5"}
Sep 11 16:17:57.908694 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.908633    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:17:57.910051 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.910027    2576 patch_prober.go:28] interesting pod/console-operator-696884fb9c-6jh9m container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.9:8443/readyz\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body=
Sep 11 16:17:57.910161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.910069    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" podUID="0f563067-de11-4c7f-92eb-0cdefa727b88" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.9:8443/readyz\": dial tcp 10.133.0.9:8443: connect: connection refused"
Sep 11 16:17:57.930819 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:57.930599    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" podStartSLOduration=4.03373691 podStartE2EDuration="6.930578672s" totalImagesPullingTime="2.896841762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:57.929727171 +0000 UTC m=+71.839528337" watchObservedRunningTime="2026-09-11 16:17:57.930578672 +0000 UTC m=+71.840379884"
Sep 11 16:17:58.096033 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.095915    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv"]
Sep 11 16:17:58.101635 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:17:58.101608    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb31607_be58_459e_b1bf_a1b96c2e68d4.slice/crio-3a2002d96869eeea2095e9e54696eca900e120b9fe3dcba35d4593bd732dde91 WatchSource:0}: Error finding container 3a2002d96869eeea2095e9e54696eca900e120b9fe3dcba35d4593bd732dde91: Status 404 returned error can't find the container with id 3a2002d96869eeea2095e9e54696eca900e120b9fe3dcba35d4593bd732dde91
Sep 11 16:17:58.104081 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.103648    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz"]
Sep 11 16:17:58.105897 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:17:58.105878    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61eeb2f0_b6c0_4286_80b8_3fc3268a8d90.slice/crio-04c91b17f75306126f071fd6c282f7ebd258fedf4ce319a49ec98404f2f14623 WatchSource:0}: Error finding container 04c91b17f75306126f071fd6c282f7ebd258fedf4ce319a49ec98404f2f14623: Status 404 returned error can't find the container with id 04c91b17f75306126f071fd6c282f7ebd258fedf4ce319a49ec98404f2f14623
Sep 11 16:17:58.623572 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.623543    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sjw2m_5e2a6297-0738-43c3-9d88-7cfa8d9474d7/node-ca/0.log"
Sep 11 16:17:58.913130 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.913045    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv" event={"ID":"6eb31607-be58-459e-b1bf-a1b96c2e68d4","Type":"ContainerStarted","Data":"3a2002d96869eeea2095e9e54696eca900e120b9fe3dcba35d4593bd732dde91"}
Sep 11 16:17:58.914786 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.914753    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dd2bv" event={"ID":"df08f2ff-be81-4cc3-966d-bc1be8960c4a","Type":"ContainerStarted","Data":"16a516faafcb95652fdbf933921cd1ed8b8e94b39000ccef0977509beff4a39e"}
Sep 11 16:17:58.914914 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.914883    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:17:58.916829 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.916805    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4gcmq" event={"ID":"74c8f10b-e859-4c3e-b95a-5f58ee099241","Type":"ContainerStarted","Data":"0fdf8767109a15c5d75ab8aee126f1ada172b2611f650fdcf70ce8432ccf5941"}
Sep 11 16:17:58.920898 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.920880    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/1.log"
Sep 11 16:17:58.921401 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.921385    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/0.log"
Sep 11 16:17:58.921470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.921420    2576 generic.go:356] "Generic (PLEG): container finished" podID="0f563067-de11-4c7f-92eb-0cdefa727b88" containerID="efd5d1db6a47091a0155d965e5c8d2880bd16dcba21af1e897691638b097f1c5" exitCode=255
Sep 11 16:17:58.921515 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.921492    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" event={"ID":"0f563067-de11-4c7f-92eb-0cdefa727b88","Type":"ContainerDied","Data":"efd5d1db6a47091a0155d965e5c8d2880bd16dcba21af1e897691638b097f1c5"}
Sep 11 16:17:58.921569 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.921556    2576 scope.go:117] "RemoveContainer" containerID="90077fcb6b72b4f306a1d1c2481605693c6136e1fd78c6b46352d885aa773ee3"
Sep 11 16:17:58.921747 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.921730    2576 scope.go:117] "RemoveContainer" containerID="efd5d1db6a47091a0155d965e5c8d2880bd16dcba21af1e897691638b097f1c5"
Sep 11 16:17:58.921917 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:58.921895    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-6jh9m_openshift-console-operator(0f563067-de11-4c7f-92eb-0cdefa727b88)\"" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" podUID="0f563067-de11-4c7f-92eb-0cdefa727b88"
Sep 11 16:17:58.923197 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.923177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz" event={"ID":"61eeb2f0-b6c0-4286-80b8-3fc3268a8d90","Type":"ContainerStarted","Data":"8187599522ed5aaa715cd0c53dc85cbbd8168f32daa1a392efc91d1ceb65db9b"}
Sep 11 16:17:58.923278 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.923212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz" event={"ID":"61eeb2f0-b6c0-4286-80b8-3fc3268a8d90","Type":"ContainerStarted","Data":"04c91b17f75306126f071fd6c282f7ebd258fedf4ce319a49ec98404f2f14623"}
Sep 11 16:17:58.979005 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:58.978927    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dd2bv" podStartSLOduration=66.993374937 podStartE2EDuration="1m11.978905426s" totalImagesPullingTime="4.985530489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:58.937470123 +0000 UTC m=+72.847271285" watchObservedRunningTime="2026-09-11 16:17:58.978905426 +0000 UTC m=+72.888706569"
Sep 11 16:17:59.003077 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:59.003037    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:17:59.003229 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:59.003126    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls podName:a7cbce59-f327-472b-a883-f88c2173e231 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:07.003110034 +0000 UTC m=+80.912911174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tmfk2" (UID: "a7cbce59-f327-472b-a883-f88c2173e231") : secret "samples-operator-tls" not found
Sep 11 16:17:59.003229 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.003032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:17:59.015689 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.015642    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hwvmz" podStartSLOduration=3.015626415 podStartE2EDuration="3.015626415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:59.014767859 +0000 UTC m=+72.924569020" watchObservedRunningTime="2026-09-11 16:17:59.015626415 +0000 UTC m=+72.925427571"
Sep 11 16:17:59.045715 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.045596    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4gcmq" podStartSLOduration=67.036025591 podStartE2EDuration="1m12.045573815s" totalImagesPullingTime="5.009548224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:59.045203951 +0000 UTC m=+72.955005113" watchObservedRunningTime="2026-09-11 16:17:59.045573815 +0000 UTC m=+72.955374977"
Sep 11 16:17:59.059993 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.059923    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:59.104320 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.104269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:59.104320 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.104322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:17:59.104927 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:59.104370    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:17:59.104927 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:59.104460    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:03.104440183 +0000 UTC m=+77.014241323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : configmap "kube-rbac-proxy" not found
Sep 11 16:17:59.104927 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:59.104493    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:59.104927 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:59.104561    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:03.104543215 +0000 UTC m=+77.014344372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:59.171749 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.171660    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:59.221839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.221805    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dttkg_52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e/serve-healthcheck-canary/0.log"
Sep 11 16:17:59.284416 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.284384    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:59.927730 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.927698    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/1.log"
Sep 11 16:17:59.928190 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.928071    2576 scope.go:117] "RemoveContainer" containerID="efd5d1db6a47091a0155d965e5c8d2880bd16dcba21af1e897691638b097f1c5"
Sep 11 16:17:59.928285 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:17:59.928261    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-6jh9m_openshift-console-operator(0f563067-de11-4c7f-92eb-0cdefa727b88)\"" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" podUID="0f563067-de11-4c7f-92eb-0cdefa727b88"
Sep 11 16:17:59.929686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.929663    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv" event={"ID":"6eb31607-be58-459e-b1bf-a1b96c2e68d4","Type":"ContainerStarted","Data":"588489db329184df4ec02d58c229d7388608daaa28faab95fd92473bf299f467"}
Sep 11 16:17:59.929786 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.929710    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv" event={"ID":"6eb31607-be58-459e-b1bf-a1b96c2e68d4","Type":"ContainerStarted","Data":"f9a775ff92bbb35f820b6ad2ddda660b266ec1bba4f0951dd7af9f4a74243d98"}
Sep 11 16:17:59.970144 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:17:59.970097    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-wdxwv" podStartSLOduration=3.812677512 podStartE2EDuration="4.970082167s" totalImagesPullingTime="1.157404655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:59.96944146 +0000 UTC m=+73.879242625" watchObservedRunningTime="2026-09-11 16:17:59.970082167 +0000 UTC m=+73.879883365"
Sep 11 16:18:00.165733 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:00.165698    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:01.619888 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:01.619846    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:18:01.620279 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:01.620257    2576 scope.go:117] "RemoveContainer" containerID="efd5d1db6a47091a0155d965e5c8d2880bd16dcba21af1e897691638b097f1c5"
Sep 11 16:18:01.620418 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:01.620399    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-6jh9m_openshift-console-operator(0f563067-de11-4c7f-92eb-0cdefa727b88)\"" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" podUID="0f563067-de11-4c7f-92eb-0cdefa727b88"
Sep 11 16:18:02.809170 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:02.809131    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:02.985868 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:02.985835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.094906 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:03.094816    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.131054 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:03.131017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:18:03.131054 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:03.131052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:18:03.131266 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:03.131151    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:03.131266 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:03.131167    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:03.131266 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:03.131217    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:11.131201851 +0000 UTC m=+85.041002991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:03.131266 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:03.131230    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:11.131223679 +0000 UTC m=+85.041024818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:03.252215 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:03.252181    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.330682 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:03.330630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.413836 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:03.413747    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.631046 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:03.631006    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.735574 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:03.735523    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:03.882142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:03.882109    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.085075 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:05.085034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.274052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:05.274015    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.511465 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:05.511431    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.689295 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:05.689259    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.877107 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:05.877026    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:06.472792 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:06.472757    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:06.695672 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:06.695609    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:07.060309 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.060272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:18:07.062832 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.062813    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cbce59-f327-472b-a883-f88c2173e231-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tmfk2\" (UID: \"a7cbce59-f327-472b-a883-f88c2173e231\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:18:07.090064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.090026    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:07.204494 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.204445    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"
Sep 11 16:18:07.290646 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.290615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:07.477302 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.477273    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:07.491354 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.491327    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2"]
Sep 11 16:18:07.581832 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.581806    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:07.692536 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.692500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:07.897096 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.897000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:07.955670 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:07.955641    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2" event={"ID":"a7cbce59-f327-472b-a883-f88c2173e231","Type":"ContainerStarted","Data":"42c70e1ffdf78f2c7a2881128ac044688597880bfb22211adc10489fb33b17de"}
Sep 11 16:18:08.223018 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:08.222982    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:08.661315 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:08.661229    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:08.878507 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:08.878467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:09.108393 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:09.108353    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:09.557278 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:09.557226    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:09.689728 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:09.689638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:09.964029 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:09.963992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2" event={"ID":"a7cbce59-f327-472b-a883-f88c2173e231","Type":"ContainerStarted","Data":"0ceac2ee6cdf9fda156e326ac9c6aab483be0f71002ab0919f3814a66ec2e870"}
Sep 11 16:18:09.964224 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:09.964052    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2" event={"ID":"a7cbce59-f327-472b-a883-f88c2173e231","Type":"ContainerStarted","Data":"6de9ac298bb8a8a71bb91369073870e8b5b13d6c8a8fc96f3c5c8cff15a4c296"}
Sep 11 16:18:09.996173 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:09.996133    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tmfk2" podStartSLOduration=17.265273858 podStartE2EDuration="18.996119159s" totalImagesPullingTime="1.730845301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:09.99457778 +0000 UTC m=+83.904378942" watchObservedRunningTime="2026-09-11 16:18:09.996119159 +0000 UTC m=+83.905920321"
Sep 11 16:18:10.286724 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:10.286619    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:10.872433 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:10.872389    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:11.073489 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:11.073437    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:11.194088 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:11.194059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:18:11.194240 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:11.194104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:18:11.194280 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:11.194268    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:11.194330 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:11.194320    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls podName:1a163692-fd52-4032-8ae6-3966412e2561 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.194306447 +0000 UTC m=+101.104107600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bbldx" (UID: "1a163692-fd52-4032-8ae6-3966412e2561") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:11.194644 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:11.194625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a163692-fd52-4032-8ae6-3966412e2561-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:18:11.466531 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:11.466448    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:11.568930 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:11.568891    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:11.693916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:11.693879    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:11.876400 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:11.876308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:12.866186 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:12.866153    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:13.075590 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:13.075541    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:13.304924 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:13.304889    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:13.478028 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:13.477997    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.057037 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:15.056994    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.158159 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:15.158127    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:15.598326 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:15.598299    2576 scope.go:117] "RemoveContainer" containerID="efd5d1db6a47091a0155d965e5c8d2880bd16dcba21af1e897691638b097f1c5"
Sep 11 16:18:15.986895 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:15.986866    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:18:15.987281 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:15.987266    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/1.log"
Sep 11 16:18:15.987336 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:15.987298    2576 generic.go:356] "Generic (PLEG): container finished" podID="0f563067-de11-4c7f-92eb-0cdefa727b88" containerID="9da24db15197828c6a90b4b70f6f1e34b3a0b6f5808a4eb556288005a029992d" exitCode=255
Sep 11 16:18:15.987371 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:15.987351    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" event={"ID":"0f563067-de11-4c7f-92eb-0cdefa727b88","Type":"ContainerDied","Data":"9da24db15197828c6a90b4b70f6f1e34b3a0b6f5808a4eb556288005a029992d"}
Sep 11 16:18:15.987406 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:15.987396    2576 scope.go:117] "RemoveContainer" containerID="efd5d1db6a47091a0155d965e5c8d2880bd16dcba21af1e897691638b097f1c5"
Sep 11 16:18:15.987699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:15.987674    2576 scope.go:117] "RemoveContainer" containerID="9da24db15197828c6a90b4b70f6f1e34b3a0b6f5808a4eb556288005a029992d"
Sep 11 16:18:15.987866 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:15.987843    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-6jh9m_openshift-console-operator(0f563067-de11-4c7f-92eb-0cdefa727b88)\"" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" podUID="0f563067-de11-4c7f-92eb-0cdefa727b88"
Sep 11 16:18:16.992898 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:16.992867    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:18:17.909030 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:17.908980    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:18:17.909325 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:17.909313    2576 scope.go:117] "RemoveContainer" containerID="9da24db15197828c6a90b4b70f6f1e34b3a0b6f5808a4eb556288005a029992d"
Sep 11 16:18:17.909471 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:17.909451    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-6jh9m_openshift-console-operator(0f563067-de11-4c7f-92eb-0cdefa727b88)\"" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" podUID="0f563067-de11-4c7f-92eb-0cdefa727b88"
Sep 11 16:18:20.205295 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.205255    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"]
Sep 11 16:18:20.209637 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.209614    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.213273 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.213249    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 16:18:20.213489 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.213463    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 16:18:20.214470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.214437    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 16:18:20.214915 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.214892    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 16:18:20.215040 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.214909    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 16:18:20.215040 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.214910    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 16:18:20.215040 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.214895    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 16:18:20.225997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.225977    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"]
Sep 11 16:18:20.265000 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.264973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-hub\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.265145 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.265025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.265145 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.265101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-ca\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.265215 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.265156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.265215 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.265181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc28f41-bce2-448e-9d4b-8b5220d335d4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.265215 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.265208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwhk\" (UniqueName: \"kubernetes.io/projected/6dc28f41-bce2-448e-9d4b-8b5220d335d4-kube-api-access-bpwhk\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.365818 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.365786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-hub\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.366040 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.365840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.366040 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.365883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-ca\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.366040 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.365908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.366040 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.365956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc28f41-bce2-448e-9d4b-8b5220d335d4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.366040 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.365980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwhk\" (UniqueName: \"kubernetes.io/projected/6dc28f41-bce2-448e-9d4b-8b5220d335d4-kube-api-access-bpwhk\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.366888 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.366864    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc28f41-bce2-448e-9d4b-8b5220d335d4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.368551 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.368508    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-ca\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.368832 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.368812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-hub\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.369020 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.368997    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.369161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.369068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6dc28f41-bce2-448e-9d4b-8b5220d335d4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.380812 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.380790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:20.385717 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.385696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwhk\" (UniqueName: \"kubernetes.io/projected/6dc28f41-bce2-448e-9d4b-8b5220d335d4-kube-api-access-bpwhk\") pod \"cluster-proxy-proxy-agent-5b9ddbf474-jzz25\" (UID: \"6dc28f41-bce2-448e-9d4b-8b5220d335d4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.535277 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.535187    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"
Sep 11 16:18:20.566763 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.566730    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:20.835000 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:20.834903    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25"]
Sep 11 16:18:20.837854 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:18:20.837831    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dc28f41_bce2_448e_9d4b_8b5220d335d4.slice/crio-86511438aa4d567899fea84f25d9ecd3f6fdffef40edf837bebcadc2a70b7bf8 WatchSource:0}: Error finding container 86511438aa4d567899fea84f25d9ecd3f6fdffef40edf837bebcadc2a70b7bf8: Status 404 returned error can't find the container with id 86511438aa4d567899fea84f25d9ecd3f6fdffef40edf837bebcadc2a70b7bf8
Sep 11 16:18:21.012646 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:21.012610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25" event={"ID":"6dc28f41-bce2-448e-9d4b-8b5220d335d4","Type":"ContainerStarted","Data":"86511438aa4d567899fea84f25d9ecd3f6fdffef40edf837bebcadc2a70b7bf8"}
Sep 11 16:18:21.619623 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:21.619560    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:18:21.620094 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:21.620054    2576 scope.go:117] "RemoveContainer" containerID="9da24db15197828c6a90b4b70f6f1e34b3a0b6f5808a4eb556288005a029992d"
Sep 11 16:18:21.620241 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:21.620215    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-6jh9m_openshift-console-operator(0f563067-de11-4c7f-92eb-0cdefa727b88)\"" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" podUID="0f563067-de11-4c7f-92eb-0cdefa727b88"
Sep 11 16:18:22.927417 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:22.927375    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:23.016363 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:23.016326    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:24.300088 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:24.300053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:18:24.302629 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:24.302597    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b250f482-a270-4c37-89a0-69fee0abc0ae-metrics-tls\") pod \"dns-default-rzth4\" (UID: \"b250f482-a270-4c37-89a0-69fee0abc0ae\") " pod="openshift-dns/dns-default-rzth4"
Sep 11 16:18:24.339494 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:24.339474    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6q2x\""
Sep 11 16:18:24.346742 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:24.346721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rzth4"
Sep 11 16:18:24.628106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:24.628080    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rzth4"]
Sep 11 16:18:24.630037 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:18:24.630002    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb250f482_a270_4c37_89a0_69fee0abc0ae.slice/crio-e6332da9154fcf349b70221c5ce870b11bb0465665522def68b22fe3f26e2db1 WatchSource:0}: Error finding container e6332da9154fcf349b70221c5ce870b11bb0465665522def68b22fe3f26e2db1: Status 404 returned error can't find the container with id e6332da9154fcf349b70221c5ce870b11bb0465665522def68b22fe3f26e2db1
Sep 11 16:18:25.028208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:25.028173    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25" event={"ID":"6dc28f41-bce2-448e-9d4b-8b5220d335d4","Type":"ContainerStarted","Data":"5930dc8508a944b77df1e08fd8e4e05728ea0d90da3ba51d4ef31e9c3e45406f"}
Sep 11 16:18:25.028404 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:25.028219    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25" event={"ID":"6dc28f41-bce2-448e-9d4b-8b5220d335d4","Type":"ContainerStarted","Data":"d691bb01daf66889d597f61c1bb3f3145383add773a919baff1d4e6973a5843e"}
Sep 11 16:18:25.028404 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:25.028237    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25" event={"ID":"6dc28f41-bce2-448e-9d4b-8b5220d335d4","Type":"ContainerStarted","Data":"a76dec3fb65389cd75c508fdde60588d125e040f278b20869d20dd7a3743f05e"}
Sep 11 16:18:25.029276 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:25.029253    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzth4" event={"ID":"b250f482-a270-4c37-89a0-69fee0abc0ae","Type":"ContainerStarted","Data":"e6332da9154fcf349b70221c5ce870b11bb0465665522def68b22fe3f26e2db1"}
Sep 11 16:18:25.057673 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:25.057627    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b9ddbf474-jzz25" podStartSLOduration=1.572111113 podStartE2EDuration="5.0576118s" totalImagesPullingTime="3.485500687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:25.054696004 +0000 UTC m=+98.964497190" watchObservedRunningTime="2026-09-11 16:18:25.0576118 +0000 UTC m=+98.967412962"
Sep 11 16:18:26.522409 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:26.522320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:26.623340 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:26.623270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:26.799614 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:26.799516    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:26.982873 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:26.982843    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:27.037120 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:27.037075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzth4" event={"ID":"b250f482-a270-4c37-89a0-69fee0abc0ae","Type":"ContainerStarted","Data":"3cec782637120daaeb8c09a61e02ead2628d9d9a48daf9031f9e37d7fb713226"}
Sep 11 16:18:27.037291 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:27.037169    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rzth4" event={"ID":"b250f482-a270-4c37-89a0-69fee0abc0ae","Type":"ContainerStarted","Data":"c50fa4ec6ae3fa70c282f1f3ab0cf73a511f42513cbefbca0de1b61d01bc7046"}
Sep 11 16:18:27.037291 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:27.037225    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rzth4"
Sep 11 16:18:27.062597 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:27.062510    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rzth4" podStartSLOduration=65.457234958 podStartE2EDuration="1m7.062496397s" totalImagesPullingTime="1.605261439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:27.061323432 +0000 UTC m=+100.971124595" watchObservedRunningTime="2026-09-11 16:18:27.062496397 +0000 UTC m=+100.972297559"
Sep 11 16:18:27.223666 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:27.223622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:18:27.226103 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:27.226078    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a163692-fd52-4032-8ae6-3966412e2561-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbldx\" (UID: \"1a163692-fd52-4032-8ae6-3966412e2561\") " pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:18:27.494235 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:27.494188    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bbldx"
Sep 11 16:18:27.777605 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:27.777531    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bbldx"]
Sep 11 16:18:27.780496 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:18:27.780472    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a163692_fd52_4032_8ae6_3966412e2561.slice/crio-bf7affd190ffae40cf46eb502677eda79f56cc713380ac858caf0e82e7512cd1 WatchSource:0}: Error finding container bf7affd190ffae40cf46eb502677eda79f56cc713380ac858caf0e82e7512cd1: Status 404 returned error can't find the container with id bf7affd190ffae40cf46eb502677eda79f56cc713380ac858caf0e82e7512cd1
Sep 11 16:18:28.042349 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:28.042270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbldx" event={"ID":"1a163692-fd52-4032-8ae6-3966412e2561","Type":"ContainerStarted","Data":"edb3dbceadc3c55d97ad3915ad242ac471e5b8182127af2aeeb55123d15fd0d2"}
Sep 11 16:18:28.042349 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:28.042323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbldx" event={"ID":"1a163692-fd52-4032-8ae6-3966412e2561","Type":"ContainerStarted","Data":"bf7affd190ffae40cf46eb502677eda79f56cc713380ac858caf0e82e7512cd1"}
Sep 11 16:18:29.048927 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:29.048894    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbldx" event={"ID":"1a163692-fd52-4032-8ae6-3966412e2561","Type":"ContainerStarted","Data":"33158563d51eb435801c947a52d117b7387cec3d68da58bddb942f2b9669909d"}
Sep 11 16:18:29.775615 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:29.775570    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:29.859186 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:29.859157    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:29.932423 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:29.932403    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dd2bv"
Sep 11 16:18:30.054275 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:30.054188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbldx" event={"ID":"1a163692-fd52-4032-8ae6-3966412e2561","Type":"ContainerStarted","Data":"9e38d2a1e55dd5ab0c440b876a60a834e4bee0b30e64634242381fb75cd9f33d"}
Sep 11 16:18:30.077631 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:30.077577    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bbldx" podStartSLOduration=33.13827138 podStartE2EDuration="35.077560254s" totalImagesPullingTime="1.939288874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:30.076957573 +0000 UTC m=+103.986758728" watchObservedRunningTime="2026-09-11 16:18:30.077560254 +0000 UTC m=+103.987361415"
Sep 11 16:18:30.408779 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:30.408694    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:32.435030 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:32.434998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:32.598335 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:32.598301    2576 scope.go:117] "RemoveContainer" containerID="9da24db15197828c6a90b4b70f6f1e34b3a0b6f5808a4eb556288005a029992d"
Sep 11 16:18:32.598515 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:18:32.598493    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-6jh9m_openshift-console-operator(0f563067-de11-4c7f-92eb-0cdefa727b88)\"" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" podUID="0f563067-de11-4c7f-92eb-0cdefa727b88"
Sep 11 16:18:32.682022 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:32.681987    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:32.766798 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:32.766763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:33.974534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:33.974497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.186619 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.186573    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.303579 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.303484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.461067 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.461035    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pgfps"]
Sep 11 16:18:34.463915 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.463896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.470287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.470244    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 16:18:34.470287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.470258    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 16:18:34.470445 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.470382    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7f2r7\""
Sep 11 16:18:34.483229 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.483212    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 16:18:34.483799 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.483783    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 16:18:34.490060 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.490047    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 16:18:34.490130 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.490049    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 16:18:34.549575 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.549546    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.571200 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.571128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.571200 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.571172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-accelerators-collector-config\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.571368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.571242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-wtmp\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.571368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.571279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-textfile\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.571368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.571301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65da7519-5eab-41d8-9009-076d71e5c154-root\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.571368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.571324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-tls\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.571368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.571347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5df\" (UniqueName: \"kubernetes.io/projected/65da7519-5eab-41d8-9009-076d71e5c154-kube-api-access-ks5df\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.571368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.571368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65da7519-5eab-41d8-9009-076d71e5c154-sys\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.571569 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.571382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65da7519-5eab-41d8-9009-076d71e5c154-metrics-client-ca\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.646409 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.646378    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.672031 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672182 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-accelerators-collector-config\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672182 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-wtmp\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672182 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-textfile\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672338 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65da7519-5eab-41d8-9009-076d71e5c154-root\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672338 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672223    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-wtmp\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672338 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-tls\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672338 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/65da7519-5eab-41d8-9009-076d71e5c154-root\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672338 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5df\" (UniqueName: \"kubernetes.io/projected/65da7519-5eab-41d8-9009-076d71e5c154-kube-api-access-ks5df\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672338 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65da7519-5eab-41d8-9009-076d71e5c154-sys\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672647 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65da7519-5eab-41d8-9009-076d71e5c154-metrics-client-ca\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672647 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672394    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65da7519-5eab-41d8-9009-076d71e5c154-sys\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672647 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672456    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-textfile\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.672915 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.672891    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-accelerators-collector-config\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.673201 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.673179    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65da7519-5eab-41d8-9009-076d71e5c154-metrics-client-ca\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.674660 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.674635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.674744 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.674723    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/65da7519-5eab-41d8-9009-076d71e5c154-node-exporter-tls\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.682208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.682189    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5df\" (UniqueName: \"kubernetes.io/projected/65da7519-5eab-41d8-9009-076d71e5c154-kube-api-access-ks5df\") pod \"node-exporter-pgfps\" (UID: \"65da7519-5eab-41d8-9009-076d71e5c154\") " pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.743596 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.743558    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.772455 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.772419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pgfps"
Sep 11 16:18:34.782997 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:18:34.782974    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65da7519_5eab_41d8_9009_076d71e5c154.slice/crio-e5cbd06f8651d9918e3a9545830cfe0dd65ade5e69b39c752b69d7af9ebea275 WatchSource:0}: Error finding container e5cbd06f8651d9918e3a9545830cfe0dd65ade5e69b39c752b69d7af9ebea275: Status 404 returned error can't find the container with id e5cbd06f8651d9918e3a9545830cfe0dd65ade5e69b39c752b69d7af9ebea275
Sep 11 16:18:34.961640 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:34.961613    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:35.070672 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:35.070637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pgfps" event={"ID":"65da7519-5eab-41d8-9009-076d71e5c154","Type":"ContainerStarted","Data":"e5cbd06f8651d9918e3a9545830cfe0dd65ade5e69b39c752b69d7af9ebea275"}
Sep 11 16:18:35.075259 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:35.075234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:35.329880 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:35.329793    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:35.425640 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:35.425585    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:35.519707 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:35.519619    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:35.859968 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:35.859906    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:35.961648 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:35.961552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:36.075660 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:36.075627    2576 generic.go:356] "Generic (PLEG): container finished" podID="65da7519-5eab-41d8-9009-076d71e5c154" containerID="123339b100d3a4ec1e65be507cdacc48a6fd2a0126f811ba700b43a9c896a4c0" exitCode=0
Sep 11 16:18:36.076103 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:36.075720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pgfps" event={"ID":"65da7519-5eab-41d8-9009-076d71e5c154","Type":"ContainerDied","Data":"123339b100d3a4ec1e65be507cdacc48a6fd2a0126f811ba700b43a9c896a4c0"}
Sep 11 16:18:37.044644 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:37.044616    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rzth4"
Sep 11 16:18:37.081470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:37.081439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pgfps" event={"ID":"65da7519-5eab-41d8-9009-076d71e5c154","Type":"ContainerStarted","Data":"168c1587998e7d3139bfb96bbdea740bfee3e77cb9184b0fdbc33f5846cff30f"}
Sep 11 16:18:37.081925 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:37.081490    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pgfps" event={"ID":"65da7519-5eab-41d8-9009-076d71e5c154","Type":"ContainerStarted","Data":"6aebc56d9c02a76ddfded7bc79b09b010fedb59217979b649b2bc407d43135c4"}
Sep 11 16:18:37.111447 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:37.111344    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pgfps" podStartSLOduration=2.118731588 podStartE2EDuration="3.111327251s" totalImagesPullingTime="922.911622ms" totalInitContainerRuntime="69.684041ms" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:37.110737866 +0000 UTC m=+111.020539029" watchObservedRunningTime="2026-09-11 16:18:37.111327251 +0000 UTC m=+111.021128413"
Sep 11 16:18:37.384536 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:37.384452    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:37.502970 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:37.502907    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:37.730459 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:37.730430    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:38.120109 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:38.120014    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:39.065077 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:39.065039    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:40.028662 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:40.028620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:40.733486 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:40.733449    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:41.995166 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:41.995123    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:42.123061 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:42.123031    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:43.290740 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:43.290707    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:43.391236 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:43.391199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:43.492846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:43.492810    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:45.056344 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:45.056308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:45.156924 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:45.156891    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.226951 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:46.226904    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.328619 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:46.328591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.599403 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:46.599338    2576 scope.go:117] "RemoveContainer" containerID="9da24db15197828c6a90b4b70f6f1e34b3a0b6f5808a4eb556288005a029992d"
Sep 11 16:18:47.115436 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.115407    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:18:47.115586 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.115481    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m" event={"ID":"0f563067-de11-4c7f-92eb-0cdefa727b88","Type":"ContainerStarted","Data":"181a414155b4c656feef1066ff258c999f2ad960999ef6a6b069745f9b83342b"}
Sep 11 16:18:47.115872 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.115840    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:18:47.120223 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.120198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.120344 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.120323    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-6jh9m"
Sep 11 16:18:47.245120 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.245082    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.452575 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.452543    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.559620 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.559582    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.651801 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.651772    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.812758 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.812677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.909400 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:47.909373    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:48.445847 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:48.445810    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.042367 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:49.042330    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.278350 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:49.278309    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.659527 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:49.659488    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.756953 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:49.756895    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.866958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:49.866904    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:55.181555 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:55.181524    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:55.282923 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:55.282896    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:56.336308 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:56.336271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:18:56.338703 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:56.338685    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45bda4f-12ee-46a5-8d83-7a1602904679-metrics-certs\") pod \"network-metrics-daemon-9sg48\" (UID: \"e45bda4f-12ee-46a5-8d83-7a1602904679\") " pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:18:56.608180 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:56.608107    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kx52\""
Sep 11 16:18:56.616447 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:56.616432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9sg48"
Sep 11 16:18:56.925815 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:56.925790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9sg48"]
Sep 11 16:18:56.927490 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:18:56.927461    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45bda4f_12ee_46a5_8d83_7a1602904679.slice/crio-ee17d7916e413d80a1f5c916d0472f9150fa53e6eca01d04b5efae3021c77732 WatchSource:0}: Error finding container ee17d7916e413d80a1f5c916d0472f9150fa53e6eca01d04b5efae3021c77732: Status 404 returned error can't find the container with id ee17d7916e413d80a1f5c916d0472f9150fa53e6eca01d04b5efae3021c77732
Sep 11 16:18:57.148908 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:57.148873    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9sg48" event={"ID":"e45bda4f-12ee-46a5-8d83-7a1602904679","Type":"ContainerStarted","Data":"ee17d7916e413d80a1f5c916d0472f9150fa53e6eca01d04b5efae3021c77732"}
Sep 11 16:18:59.158237 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:59.158198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9sg48" event={"ID":"e45bda4f-12ee-46a5-8d83-7a1602904679","Type":"ContainerStarted","Data":"ced9f1f4ea2e31e65f321f9bcb5b14350b901375656d743886ec5b258f08519c"}
Sep 11 16:18:59.158684 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:59.158262    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9sg48" event={"ID":"e45bda4f-12ee-46a5-8d83-7a1602904679","Type":"ContainerStarted","Data":"e5961c4389972ae8a0e9e187c8743c7b0c95079bd9c4b3ea02ff0c7f6209c7fe"}
Sep 11 16:18:59.178390 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:59.178340    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9sg48" podStartSLOduration=130.838925949 podStartE2EDuration="2m12.178326299s" totalImagesPullingTime="1.33940035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:59.17682446 +0000 UTC m=+133.086625622" watchObservedRunningTime="2026-09-11 16:18:59.178326299 +0000 UTC m=+133.088127460"
Sep 11 16:18:59.321420 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:59.321381    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:59.412586 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:18:59.412500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:00.059263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:00.059224    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:00.129954 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:00.129899    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:03.661874 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:03.661840    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:03.734031 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:03.733994    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:04.213210 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:04.213173    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:04.349858 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:04.349822    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:04.427888 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:04.427857    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:07.844438 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:07.844396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:07.945043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:07.945011    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:11.571241 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:11.571198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:11.664111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:11.664068    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:12.437329 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:12.437294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:12.512761 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:12.512730    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:13.045168 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:13.045138    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:13.142262 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:13.142216    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:15.056088 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:15.056052    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:15.156836 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:15.156800    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:17.326959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:17.326894    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:17.452838 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:17.452746    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:21.053868 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:21.053830    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:21.137721 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:21.137661    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:21.295443 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:21.295412    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:21.396347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:21.396262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:26.257184 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:26.257150    2576 generic.go:356] "Generic (PLEG): container finished" podID="9065a2bc-304a-44a4-ab32-fe1cd9e41a59" containerID="2ec32b585d83a9dd160a6e31e099d69e3917254c01b6c46b1c2677a25048a667" exitCode=0
Sep 11 16:19:26.257562 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:26.257192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r" event={"ID":"9065a2bc-304a-44a4-ab32-fe1cd9e41a59","Type":"ContainerDied","Data":"2ec32b585d83a9dd160a6e31e099d69e3917254c01b6c46b1c2677a25048a667"}
Sep 11 16:19:26.257562 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:26.257503    2576 scope.go:117] "RemoveContainer" containerID="2ec32b585d83a9dd160a6e31e099d69e3917254c01b6c46b1c2677a25048a667"
Sep 11 16:19:27.262829 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:27.262793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-htl5r" event={"ID":"9065a2bc-304a-44a4-ab32-fe1cd9e41a59","Type":"ContainerStarted","Data":"56306aefabdfa83f1686335bc1aa020c86ee62191278097cdc995df481fb5134"}
Sep 11 16:19:30.055983 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:30.055929    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:30.136038 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:30.135999    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:31.902616 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:31.902566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:50.569835 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:50.569753    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:50.644866 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:19:50.644832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:00.778984 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:00.778928    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:00.879765 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:00.879709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:00.958074 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:00.958035    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:01.534129 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:01.534090    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:01.711653 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:01.711613    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:02.549973 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:02.549919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:02.620997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:02.620960    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:11.713388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:11.713342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:11.811865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:11.811829    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:11.887865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:11.887831    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:15.058460 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:15.058416    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:15.160065 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:15.160032    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:34.945674 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:34.945628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:35.018285 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:35.018251    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:39.420364 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:20:39.420333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:18.997496 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:18.997427    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:37.945007 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:37.944971    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:46.456116 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:46.456084    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:21:46.457841 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:46.457816    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:21:46.462963 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:46.462926    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:21:46.465158 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:46.465139    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:21:46.466657 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:46.466639    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:21:53.175899 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:53.175862    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:53.279793 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:53.279761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:53.379763 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:21:53.379729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:10.782350 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:10.782314    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:10.850563 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:10.850524    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:15.737998 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:15.737961    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:15.829756 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:15.829728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:21.335032 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:21.334987    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:22.508003 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:22.507956    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:22.609801 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:22.609769    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:22.713110 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:22.713074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:22.809528 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:22.809430    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:23.115553 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:23.115468    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:23.275264 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:23.275231    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:23.434228 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:23.434140    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:23.535828 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:22:23.535794    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:04.302417 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:04.302318    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:04.368821 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:04.368786    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:15.381616 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:15.381582    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:15.481096 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:15.481059    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:46.842252 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:46.842214    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:46.958955 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:46.958899    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:46.964186 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:46.964162    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-f2rwj"]
Sep 11 16:23:46.967019 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:46.967001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:46.973623 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:46.973601    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:23:46.973715 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:46.973629    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:23:46.974736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:46.974712    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-6ptqk\""
Sep 11 16:23:46.974736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:46.974726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 11 16:23:46.981349 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:46.981331    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-f2rwj"]
Sep 11 16:23:47.016924 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.016893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvnjf\" (UniqueName: \"kubernetes.io/projected/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-kube-api-access-zvnjf\") pod \"kserve-controller-manager-656b75fc8d-f2rwj\" (UID: \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\") " pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:47.017058 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.016971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-cert\") pod \"kserve-controller-manager-656b75fc8d-f2rwj\" (UID: \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\") " pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:47.061802 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.061771    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-wc8vk"]
Sep 11 16:23:47.064762 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.064747    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:47.067760 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.067737    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 11 16:23:47.068198 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.068183    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9d2b\""
Sep 11 16:23:47.071781 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.071747    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:47.100452 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.100379    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wc8vk"]
Sep 11 16:23:47.117733 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.117703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxjs\" (UniqueName: \"kubernetes.io/projected/3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d-kube-api-access-tsxjs\") pod \"seaweedfs-64568bcd49-wc8vk\" (UID: \"3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d\") " pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:47.117871 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.117750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d-data\") pod \"seaweedfs-64568bcd49-wc8vk\" (UID: \"3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d\") " pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:47.117871 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.117807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-cert\") pod \"kserve-controller-manager-656b75fc8d-f2rwj\" (UID: \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\") " pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:47.118016 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.117868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvnjf\" (UniqueName: \"kubernetes.io/projected/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-kube-api-access-zvnjf\") pod \"kserve-controller-manager-656b75fc8d-f2rwj\" (UID: \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\") " pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:47.118016 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:23:47.117961    2576 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 11 16:23:47.118119 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:23:47.118034    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-cert podName:a1dfcf00-2a60-4dac-9722-73c8d446ceb4 nodeName:}" failed. No retries permitted until 2026-09-11 16:23:47.618017876 +0000 UTC m=+421.527819017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-cert") pod "kserve-controller-manager-656b75fc8d-f2rwj" (UID: "a1dfcf00-2a60-4dac-9722-73c8d446ceb4") : secret "kserve-webhook-server-cert" not found
Sep 11 16:23:47.133187 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.133160    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvnjf\" (UniqueName: \"kubernetes.io/projected/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-kube-api-access-zvnjf\") pod \"kserve-controller-manager-656b75fc8d-f2rwj\" (UID: \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\") " pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:47.182283 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.182254    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:47.218787 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.218756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxjs\" (UniqueName: \"kubernetes.io/projected/3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d-kube-api-access-tsxjs\") pod \"seaweedfs-64568bcd49-wc8vk\" (UID: \"3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d\") " pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:47.218787 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.218794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d-data\") pod \"seaweedfs-64568bcd49-wc8vk\" (UID: \"3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d\") " pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:47.219143 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.219127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d-data\") pod \"seaweedfs-64568bcd49-wc8vk\" (UID: \"3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d\") " pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:47.232762 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.232729    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxjs\" (UniqueName: \"kubernetes.io/projected/3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d-kube-api-access-tsxjs\") pod \"seaweedfs-64568bcd49-wc8vk\" (UID: \"3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d\") " pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:47.312281 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.312247    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:47.373233 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.373157    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:47.621265 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.621218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-cert\") pod \"kserve-controller-manager-656b75fc8d-f2rwj\" (UID: \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\") " pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:47.623982 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.623903    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-cert\") pod \"kserve-controller-manager-656b75fc8d-f2rwj\" (UID: \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\") " pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:47.648574 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.648549    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wc8vk"]
Sep 11 16:23:47.651380 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:23:47.651356    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a626c9e_ba00_4e72_8cf9_ed7b0f857a5d.slice/crio-b6195e4d56d8e7be8eff655dcc39311e97389a9f8077f3328036ee2efc0a4837 WatchSource:0}: Error finding container b6195e4d56d8e7be8eff655dcc39311e97389a9f8077f3328036ee2efc0a4837: Status 404 returned error can't find the container with id b6195e4d56d8e7be8eff655dcc39311e97389a9f8077f3328036ee2efc0a4837
Sep 11 16:23:47.652620 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.652605    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:23:47.877296 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.877217    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:47.935240 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:47.935214    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:48.039608 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:48.039552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:48.125058 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:48.125026    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:48.128349 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:48.128248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wc8vk" event={"ID":"3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d","Type":"ContainerStarted","Data":"b6195e4d56d8e7be8eff655dcc39311e97389a9f8077f3328036ee2efc0a4837"}
Sep 11 16:23:48.161397 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:48.161368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-f2rwj"]
Sep 11 16:23:48.202592 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:23:48.202560    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1dfcf00_2a60_4dac_9722_73c8d446ceb4.slice/crio-a16dcf539a15e04bf7c0d5729c811071d22025719f7e80bdc537a23a45fd3ff7 WatchSource:0}: Error finding container a16dcf539a15e04bf7c0d5729c811071d22025719f7e80bdc537a23a45fd3ff7: Status 404 returned error can't find the container with id a16dcf539a15e04bf7c0d5729c811071d22025719f7e80bdc537a23a45fd3ff7
Sep 11 16:23:49.133968 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:49.133880    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj" event={"ID":"a1dfcf00-2a60-4dac-9722-73c8d446ceb4","Type":"ContainerStarted","Data":"a16dcf539a15e04bf7c0d5729c811071d22025719f7e80bdc537a23a45fd3ff7"}
Sep 11 16:23:52.147214 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:52.147181    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj" event={"ID":"a1dfcf00-2a60-4dac-9722-73c8d446ceb4","Type":"ContainerStarted","Data":"c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d"}
Sep 11 16:23:52.147640 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:52.147256    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:23:52.148703 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:52.148680    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wc8vk" event={"ID":"3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d","Type":"ContainerStarted","Data":"dd68b592c3a41f08094fa34da939f764cb6339c08f0982b08a7523946a6e017c"}
Sep 11 16:23:52.148811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:52.148794    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:52.185399 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:52.185359    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj" podStartSLOduration=2.741358602 podStartE2EDuration="6.18534544s" totalImagesPullingTime="3.443986838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:52.184689042 +0000 UTC m=+426.094490204" watchObservedRunningTime="2026-09-11 16:23:52.18534544 +0000 UTC m=+426.095146602"
Sep 11 16:23:58.154619 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:58.154589    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-wc8vk"
Sep 11 16:23:58.177926 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:23:58.177877    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-wc8vk" podStartSLOduration=7.132814654 podStartE2EDuration="11.177859855s" totalImagesPullingTime="4.045045201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:52.236404446 +0000 UTC m=+426.146205608" watchObservedRunningTime="2026-09-11 16:23:58.177859855 +0000 UTC m=+432.087661016"
Sep 11 16:24:08.904741 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:08.904645    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:09.004332 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:09.004290    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:09.105266 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:09.105235    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:09.763836 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:09.763799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:19.791052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:19.791017    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:19.892729 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:19.892700    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:23.157687 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:23.157657    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:24:30.766317 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:30.766277    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-f2rwj"]
Sep 11 16:24:30.766779 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:30.766518    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj" podUID="a1dfcf00-2a60-4dac-9722-73c8d446ceb4" containerName="manager" containerID="cri-o://c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d" gracePeriod=10
Sep 11 16:24:30.801472 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:30.801445    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-lkvx9"]
Sep 11 16:24:30.804277 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:30.804261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:24:30.820586 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:30.820561    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-lkvx9"]
Sep 11 16:24:30.930847 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:30.930813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwcq\" (UniqueName: \"kubernetes.io/projected/d98ac13e-000e-4a57-8f4b-ab46bbd50895-kube-api-access-rhwcq\") pod \"kserve-controller-manager-656b75fc8d-lkvx9\" (UID: \"d98ac13e-000e-4a57-8f4b-ab46bbd50895\") " pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:24:30.931026 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:30.930871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d98ac13e-000e-4a57-8f4b-ab46bbd50895-cert\") pod \"kserve-controller-manager-656b75fc8d-lkvx9\" (UID: \"d98ac13e-000e-4a57-8f4b-ab46bbd50895\") " pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:24:31.008599 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.008578    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:24:31.031644 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.031576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d98ac13e-000e-4a57-8f4b-ab46bbd50895-cert\") pod \"kserve-controller-manager-656b75fc8d-lkvx9\" (UID: \"d98ac13e-000e-4a57-8f4b-ab46bbd50895\") " pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:24:31.031774 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.031657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwcq\" (UniqueName: \"kubernetes.io/projected/d98ac13e-000e-4a57-8f4b-ab46bbd50895-kube-api-access-rhwcq\") pod \"kserve-controller-manager-656b75fc8d-lkvx9\" (UID: \"d98ac13e-000e-4a57-8f4b-ab46bbd50895\") " pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:24:31.034498 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.034474    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d98ac13e-000e-4a57-8f4b-ab46bbd50895-cert\") pod \"kserve-controller-manager-656b75fc8d-lkvx9\" (UID: \"d98ac13e-000e-4a57-8f4b-ab46bbd50895\") " pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:24:31.044586 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.044562    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwcq\" (UniqueName: \"kubernetes.io/projected/d98ac13e-000e-4a57-8f4b-ab46bbd50895-kube-api-access-rhwcq\") pod \"kserve-controller-manager-656b75fc8d-lkvx9\" (UID: \"d98ac13e-000e-4a57-8f4b-ab46bbd50895\") " pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:24:31.132470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.132434    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvnjf\" (UniqueName: \"kubernetes.io/projected/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-kube-api-access-zvnjf\") pod \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\" (UID: \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\") "
Sep 11 16:24:31.132470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.132474    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-cert\") pod \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\" (UID: \"a1dfcf00-2a60-4dac-9722-73c8d446ceb4\") "
Sep 11 16:24:31.134742 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.134717    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-cert" (OuterVolumeSpecName: "cert") pod "a1dfcf00-2a60-4dac-9722-73c8d446ceb4" (UID: "a1dfcf00-2a60-4dac-9722-73c8d446ceb4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:24:31.134742 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.134730    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-kube-api-access-zvnjf" (OuterVolumeSpecName: "kube-api-access-zvnjf") pod "a1dfcf00-2a60-4dac-9722-73c8d446ceb4" (UID: "a1dfcf00-2a60-4dac-9722-73c8d446ceb4"). InnerVolumeSpecName "kube-api-access-zvnjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:24:31.145899 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.145875    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:24:31.233404 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.233370    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvnjf\" (UniqueName: \"kubernetes.io/projected/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-kube-api-access-zvnjf\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:24:31.233404 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.233397    2576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1dfcf00-2a60-4dac-9722-73c8d446ceb4-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:24:31.293042 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.292927    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1dfcf00-2a60-4dac-9722-73c8d446ceb4" containerID="c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d" exitCode=0
Sep 11 16:24:31.293042 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.293000    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj"
Sep 11 16:24:31.293239 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.293037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj" event={"ID":"a1dfcf00-2a60-4dac-9722-73c8d446ceb4","Type":"ContainerDied","Data":"c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d"}
Sep 11 16:24:31.293239 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.293087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-f2rwj" event={"ID":"a1dfcf00-2a60-4dac-9722-73c8d446ceb4","Type":"ContainerDied","Data":"a16dcf539a15e04bf7c0d5729c811071d22025719f7e80bdc537a23a45fd3ff7"}
Sep 11 16:24:31.293239 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.293110    2576 scope.go:117] "RemoveContainer" containerID="c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d"
Sep 11 16:24:31.301149 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.301130    2576 scope.go:117] "RemoveContainer" containerID="c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d"
Sep 11 16:24:31.301429 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:24:31.301407    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d\": container with ID starting with c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d not found: ID does not exist" containerID="c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d"
Sep 11 16:24:31.301492 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.301442    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d"} err="failed to get container status \"c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d\": rpc error: code = NotFound desc = could not find container \"c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d\": container with ID starting with c72e64b76c20b4fda64a156515b429f429b0a151bbbfd418fcf9a7556a35ff2d not found: ID does not exist"
Sep 11 16:24:31.317839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.317817    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-f2rwj"]
Sep 11 16:24:31.324861 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.324842    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-f2rwj"]
Sep 11 16:24:31.413278 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:31.413251    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-lkvx9"]
Sep 11 16:24:31.416649 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:24:31.416627    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98ac13e_000e_4a57_8f4b_ab46bbd50895.slice/crio-e25f84cfb553976a8247da31c1e8c1e22c6eb25640ef041031aff9b8f1a91e74 WatchSource:0}: Error finding container e25f84cfb553976a8247da31c1e8c1e22c6eb25640ef041031aff9b8f1a91e74: Status 404 returned error can't find the container with id e25f84cfb553976a8247da31c1e8c1e22c6eb25640ef041031aff9b8f1a91e74
Sep 11 16:24:32.298970 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:32.298918    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9" event={"ID":"d98ac13e-000e-4a57-8f4b-ab46bbd50895","Type":"ContainerStarted","Data":"b338da5f44605a8054422a2c2eea9d4a8c3e11da7ed10587ea8875e0cc109594"}
Sep 11 16:24:32.299375 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:32.298993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9" event={"ID":"d98ac13e-000e-4a57-8f4b-ab46bbd50895","Type":"ContainerStarted","Data":"e25f84cfb553976a8247da31c1e8c1e22c6eb25640ef041031aff9b8f1a91e74"}
Sep 11 16:24:32.299375 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:32.299036    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:24:32.322377 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:32.322321    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9" podStartSLOduration=1.9116675600000002 podStartE2EDuration="2.32229953s" totalImagesPullingTime="410.63197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:24:32.322005654 +0000 UTC m=+466.231806816" watchObservedRunningTime="2026-09-11 16:24:32.32229953 +0000 UTC m=+466.232100689"
Sep 11 16:24:32.602802 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:24:32.602722    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1dfcf00-2a60-4dac-9722-73c8d446ceb4" path="/var/lib/kubelet/pods/a1dfcf00-2a60-4dac-9722-73c8d446ceb4/volumes"
Sep 11 16:25:03.307214 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:03.307182    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-656b75fc8d-lkvx9"
Sep 11 16:25:04.778122 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.778087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:04.874246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.874213    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-jdqhw"]
Sep 11 16:25:04.874541 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.874525    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1dfcf00-2a60-4dac-9722-73c8d446ceb4" containerName="manager"
Sep 11 16:25:04.874619 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.874545    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1dfcf00-2a60-4dac-9722-73c8d446ceb4" containerName="manager"
Sep 11 16:25:04.874676 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.874626    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1dfcf00-2a60-4dac-9722-73c8d446ceb4" containerName="manager"
Sep 11 16:25:04.877475 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.877455    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:25:04.881258 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.881240    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-vbz9f\""
Sep 11 16:25:04.882347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.882325    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 11 16:25:04.895558 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.895526    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:04.896224 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.896205    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-nl2xp"]
Sep 11 16:25:04.899134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.899120    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:04.915567 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.915542    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 11 16:25:04.915691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.915586    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 11 16:25:04.915922 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.915909    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-cc652\""
Sep 11 16:25:04.921565 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.921545    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jdqhw"]
Sep 11 16:25:04.926368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.926345    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nl2xp"]
Sep 11 16:25:04.963677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.963650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blv7g\" (UniqueName: \"kubernetes.io/projected/59abc94d-ea3c-4e55-9faf-ccba8939ec71-kube-api-access-blv7g\") pod \"model-serving-api-7b954ffc95-jdqhw\" (UID: \"59abc94d-ea3c-4e55-9faf-ccba8939ec71\") " pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:25:04.963827 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.963682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjcm\" (UniqueName: \"kubernetes.io/projected/5751c273-e764-4438-b5f9-9975a0e07e3b-kube-api-access-znjcm\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:04.963827 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.963708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59abc94d-ea3c-4e55-9faf-ccba8939ec71-tls-certs\") pod \"model-serving-api-7b954ffc95-jdqhw\" (UID: \"59abc94d-ea3c-4e55-9faf-ccba8939ec71\") " pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:25:04.963827 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.963723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5751c273-e764-4438-b5f9-9975a0e07e3b-metrics-certs\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:04.963827 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:04.963788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5751c273-e764-4438-b5f9-9975a0e07e3b-cert\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:05.025536 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.025504    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:05.064641 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.064552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5751c273-e764-4438-b5f9-9975a0e07e3b-cert\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:05.064641 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.064593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blv7g\" (UniqueName: \"kubernetes.io/projected/59abc94d-ea3c-4e55-9faf-ccba8939ec71-kube-api-access-blv7g\") pod \"model-serving-api-7b954ffc95-jdqhw\" (UID: \"59abc94d-ea3c-4e55-9faf-ccba8939ec71\") " pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:25:05.064641 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.064611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjcm\" (UniqueName: \"kubernetes.io/projected/5751c273-e764-4438-b5f9-9975a0e07e3b-kube-api-access-znjcm\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:05.064853 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.064737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59abc94d-ea3c-4e55-9faf-ccba8939ec71-tls-certs\") pod \"model-serving-api-7b954ffc95-jdqhw\" (UID: \"59abc94d-ea3c-4e55-9faf-ccba8939ec71\") " pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:25:05.064853 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.064765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5751c273-e764-4438-b5f9-9975a0e07e3b-metrics-certs\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:05.064916 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:25:05.064870    2576 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 11 16:25:05.064982 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:25:05.064966    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5751c273-e764-4438-b5f9-9975a0e07e3b-metrics-certs podName:5751c273-e764-4438-b5f9-9975a0e07e3b nodeName:}" failed. No retries permitted until 2026-09-11 16:25:05.564916292 +0000 UTC m=+499.474717435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5751c273-e764-4438-b5f9-9975a0e07e3b-metrics-certs") pod "odh-model-controller-7b966d7984-nl2xp" (UID: "5751c273-e764-4438-b5f9-9975a0e07e3b") : secret "odh-model-controller-metrics-tls" not found
Sep 11 16:25:05.067175 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.067139    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5751c273-e764-4438-b5f9-9975a0e07e3b-cert\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:05.067288 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.067267    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59abc94d-ea3c-4e55-9faf-ccba8939ec71-tls-certs\") pod \"model-serving-api-7b954ffc95-jdqhw\" (UID: \"59abc94d-ea3c-4e55-9faf-ccba8939ec71\") " pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:25:05.076997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.076967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjcm\" (UniqueName: \"kubernetes.io/projected/5751c273-e764-4438-b5f9-9975a0e07e3b-kube-api-access-znjcm\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:05.077092 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.077055    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blv7g\" (UniqueName: \"kubernetes.io/projected/59abc94d-ea3c-4e55-9faf-ccba8939ec71-kube-api-access-blv7g\") pod \"model-serving-api-7b954ffc95-jdqhw\" (UID: \"59abc94d-ea3c-4e55-9faf-ccba8939ec71\") " pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:25:05.187163 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.187123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:25:05.209583 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.209525    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:05.466778 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.466754    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jdqhw"]
Sep 11 16:25:05.469144 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:25:05.469113    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59abc94d_ea3c_4e55_9faf_ccba8939ec71.slice/crio-f1f71940c8398009df176339828e471dc3780715215b677aa0a41e1641b1fb78 WatchSource:0}: Error finding container f1f71940c8398009df176339828e471dc3780715215b677aa0a41e1641b1fb78: Status 404 returned error can't find the container with id f1f71940c8398009df176339828e471dc3780715215b677aa0a41e1641b1fb78
Sep 11 16:25:05.567094 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.567060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5751c273-e764-4438-b5f9-9975a0e07e3b-metrics-certs\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:05.569577 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.569549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5751c273-e764-4438-b5f9-9975a0e07e3b-metrics-certs\") pod \"odh-model-controller-7b966d7984-nl2xp\" (UID: \"5751c273-e764-4438-b5f9-9975a0e07e3b\") " pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:05.808452 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:05.808354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:06.087148 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:06.087059    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nl2xp"]
Sep 11 16:25:06.090088 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:25:06.090057    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5751c273_e764_4438_b5f9_9975a0e07e3b.slice/crio-8f78dba46ec20e58737c9f3d476740ce642c68b3ca04c693859c6e8d3a9534d9 WatchSource:0}: Error finding container 8f78dba46ec20e58737c9f3d476740ce642c68b3ca04c693859c6e8d3a9534d9: Status 404 returned error can't find the container with id 8f78dba46ec20e58737c9f3d476740ce642c68b3ca04c693859c6e8d3a9534d9
Sep 11 16:25:06.421696 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:06.421599    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nl2xp" event={"ID":"5751c273-e764-4438-b5f9-9975a0e07e3b","Type":"ContainerStarted","Data":"8f78dba46ec20e58737c9f3d476740ce642c68b3ca04c693859c6e8d3a9534d9"}
Sep 11 16:25:06.423113 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:06.423084    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jdqhw" event={"ID":"59abc94d-ea3c-4e55-9faf-ccba8939ec71","Type":"ContainerStarted","Data":"f1f71940c8398009df176339828e471dc3780715215b677aa0a41e1641b1fb78"}
Sep 11 16:25:06.752910 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:06.752876    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:06.840881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:06.840847    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:10.440588 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:10.440542    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nl2xp" event={"ID":"5751c273-e764-4438-b5f9-9975a0e07e3b","Type":"ContainerStarted","Data":"4dc9212f59cc5189a50e1d722423d9d7f8f2e2c0c5d986771b652c05527a7d81"}
Sep 11 16:25:10.441052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:10.440677    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:10.442072 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:10.442052    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jdqhw" event={"ID":"59abc94d-ea3c-4e55-9faf-ccba8939ec71","Type":"ContainerStarted","Data":"59d3e9e2e4dae98c082705cec0ccca59cb9bcbbc730475c3a4f137c9ebb3d59c"}
Sep 11 16:25:10.442182 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:10.442102    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:25:10.463818 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:10.463766    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-nl2xp" podStartSLOduration=3.016506881 podStartE2EDuration="6.463754251s" totalImagesPullingTime="3.44724737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:25:10.462207016 +0000 UTC m=+504.372008178" watchObservedRunningTime="2026-09-11 16:25:10.463754251 +0000 UTC m=+504.373555413"
Sep 11 16:25:10.489432 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:10.489388    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-jdqhw" podStartSLOduration=2.465835071 podStartE2EDuration="6.489377489s" totalImagesPullingTime="4.023542418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:25:10.487402134 +0000 UTC m=+504.397203296" watchObservedRunningTime="2026-09-11 16:25:10.489377489 +0000 UTC m=+504.399178701"
Sep 11 16:25:21.447518 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:21.447485    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-nl2xp"
Sep 11 16:25:21.449624 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:25:21.449602    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-jdqhw"
Sep 11 16:26:11.959971 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:26:11.959873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:42.730558 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:26:42.730529    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:46.479534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:26:46.479507    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:26:46.480087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:26:46.479554    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:26:46.486043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:26:46.486018    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:26:46.486163 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:26:46.486018    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:27:17.289670 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.289576    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"]
Sep 11 16:27:17.293001 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.292982    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.295997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.295979    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 11 16:27:17.296472 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.296457    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 11 16:27:17.305039 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.305017    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"]
Sep 11 16:27:17.452958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.452893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c881b739-c8c7-474e-94a6-3e436b6aa044-data\") pod \"seaweedfs-tls-custom-8b44d767d-mcd7s\" (UID: \"c881b739-c8c7-474e-94a6-3e436b6aa044\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.452958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.452958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zqd\" (UniqueName: \"kubernetes.io/projected/c881b739-c8c7-474e-94a6-3e436b6aa044-kube-api-access-76zqd\") pod \"seaweedfs-tls-custom-8b44d767d-mcd7s\" (UID: \"c881b739-c8c7-474e-94a6-3e436b6aa044\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.453170 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.453021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c881b739-c8c7-474e-94a6-3e436b6aa044-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-mcd7s\" (UID: \"c881b739-c8c7-474e-94a6-3e436b6aa044\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.553816 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.553737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c881b739-c8c7-474e-94a6-3e436b6aa044-data\") pod \"seaweedfs-tls-custom-8b44d767d-mcd7s\" (UID: \"c881b739-c8c7-474e-94a6-3e436b6aa044\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.553816 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.553776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zqd\" (UniqueName: \"kubernetes.io/projected/c881b739-c8c7-474e-94a6-3e436b6aa044-kube-api-access-76zqd\") pod \"seaweedfs-tls-custom-8b44d767d-mcd7s\" (UID: \"c881b739-c8c7-474e-94a6-3e436b6aa044\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.553816 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.553808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c881b739-c8c7-474e-94a6-3e436b6aa044-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-mcd7s\" (UID: \"c881b739-c8c7-474e-94a6-3e436b6aa044\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.554233 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.554215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c881b739-c8c7-474e-94a6-3e436b6aa044-data\") pod \"seaweedfs-tls-custom-8b44d767d-mcd7s\" (UID: \"c881b739-c8c7-474e-94a6-3e436b6aa044\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.556448 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.556426    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c881b739-c8c7-474e-94a6-3e436b6aa044-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-mcd7s\" (UID: \"c881b739-c8c7-474e-94a6-3e436b6aa044\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.564196 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.564173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zqd\" (UniqueName: \"kubernetes.io/projected/c881b739-c8c7-474e-94a6-3e436b6aa044-kube-api-access-76zqd\") pod \"seaweedfs-tls-custom-8b44d767d-mcd7s\" (UID: \"c881b739-c8c7-474e-94a6-3e436b6aa044\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.601318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.601271    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"
Sep 11 16:27:17.872530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.872482    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-mcd7s"]
Sep 11 16:27:17.874735 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:27:17.874712    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc881b739_c8c7_474e_94a6_3e436b6aa044.slice/crio-208ac37d2fbfbc9ce9f2fd40d6759d586594520993f0026279d8d7c1c17b7ee1 WatchSource:0}: Error finding container 208ac37d2fbfbc9ce9f2fd40d6759d586594520993f0026279d8d7c1c17b7ee1: Status 404 returned error can't find the container with id 208ac37d2fbfbc9ce9f2fd40d6759d586594520993f0026279d8d7c1c17b7ee1
Sep 11 16:27:17.902580 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:17.902545    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s" event={"ID":"c881b739-c8c7-474e-94a6-3e436b6aa044","Type":"ContainerStarted","Data":"208ac37d2fbfbc9ce9f2fd40d6759d586594520993f0026279d8d7c1c17b7ee1"}
Sep 11 16:27:18.907512 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:18.907475    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s" event={"ID":"c881b739-c8c7-474e-94a6-3e436b6aa044","Type":"ContainerStarted","Data":"b88950e946fb9033cf0c7e23a7d7b761525ec5a2438018a0ed08ba7e96fd9179"}
Sep 11 16:27:18.927171 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:18.927109    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-mcd7s" podStartSLOduration=1.6776951439999999 podStartE2EDuration="1.927090512s" totalImagesPullingTime="249.395368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:18.925669658 +0000 UTC m=+632.835470812" watchObservedRunningTime="2026-09-11 16:27:18.927090512 +0000 UTC m=+632.836891670"
Sep 11 16:27:31.840670 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:31.840626    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:33.639979 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:33.639919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:46.676135 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:46.676105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:46.776537 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:46.776504    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:58.714072 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.714029    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:58.836373 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.836338    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"]
Sep 11 16:27:58.839917 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.839898    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.842847 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.842826    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 11 16:27:58.842995 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.842832    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:27:58.843139 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.843108    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:27:58.843139 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.843123    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:27:58.843357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.843207    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:27:58.843357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.843232    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 11 16:27:58.843357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.843277    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:27:58.852703 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.852674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.852832 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.852802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.852897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.852833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7x7\" (UniqueName: \"kubernetes.io/projected/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-api-access-2w7x7\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.852897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.852860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.852897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.852887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.853138 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.853122    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"]
Sep 11 16:27:58.953917 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.953885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.954142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.953924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7x7\" (UniqueName: \"kubernetes.io/projected/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-api-access-2w7x7\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.954142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.953983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.954142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.954004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.954142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.954049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.954476 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.954450    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.954643 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.954623    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.954693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.954668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.956973 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.956931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:58.966848 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:58.966770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7x7\" (UniqueName: \"kubernetes.io/projected/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-api-access-2w7x7\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:59.006544 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:59.006491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:59.106318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:59.106282    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:59.150193 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:59.150155    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:27:59.273520 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:59.273434    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:59.434145 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:27:59.434051    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"]
Sep 11 16:27:59.437730 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:27:59.437699    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5a4ad6_f121_40d3_882a_466e3a23bb4b.slice/crio-5c9212a67b41619675e9afacbb11f4adfb6233622a309cbb9173562b85117093 WatchSource:0}: Error finding container 5c9212a67b41619675e9afacbb11f4adfb6233622a309cbb9173562b85117093: Status 404 returned error can't find the container with id 5c9212a67b41619675e9afacbb11f4adfb6233622a309cbb9173562b85117093
Sep 11 16:28:00.054415 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:00.054381    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerStarted","Data":"5c9212a67b41619675e9afacbb11f4adfb6233622a309cbb9173562b85117093"}
Sep 11 16:28:03.069562 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:03.069473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerStarted","Data":"2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46"}
Sep 11 16:28:04.615037 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:04.615004    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:07.085663 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:07.085581    2576 generic.go:356] "Generic (PLEG): container finished" podID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerID="2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46" exitCode=0
Sep 11 16:28:07.086053 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:07.085653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerDied","Data":"2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46"}
Sep 11 16:28:08.697276 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:08.696399    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:13.907409 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:13.907312    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:14.050395 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:14.050348    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:15.524179 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:15.523960    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:21.156849 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:21.156819    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerStarted","Data":"4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26"}
Sep 11 16:28:24.172556 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:24.172514    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerStarted","Data":"7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa"}
Sep 11 16:28:26.183993 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:26.183954    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerStarted","Data":"04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b"}
Sep 11 16:28:26.184459 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:26.184113    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:28:26.207221 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:26.207157    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podStartSLOduration=2.004524831 podStartE2EDuration="28.207135107s" totalImagesPullingTime="22.147288499s" totalInitContainerRuntime="4.055321777s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:58 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:26.206657955 +0000 UTC m=+700.116459119" watchObservedRunningTime="2026-09-11 16:28:26.207135107 +0000 UTC m=+700.116936278"
Sep 11 16:28:27.187804 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:27.187755    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:28:27.188206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:27.187838    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:28:27.189184 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:27.189157    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:28:27.189833 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:27.189805    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:28:28.192247 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:28.192189    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:28:28.192778 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:28.192679    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:28:28.197396 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:28.197374    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:28:29.196350 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:29.196311    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:28:29.196794 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:29.196633    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:28:39.196997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:39.196875    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:28:39.197393 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:39.197360    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:28:49.197145 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:49.197094    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:28:49.197639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:49.197560    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:28:56.798892 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:56.798856    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:59.197134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:59.197087    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:28:59.197596 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:28:59.197497    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:29:09.197039 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:09.196988    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:29:09.197523 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:09.197346    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:29:19.196691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:19.196641    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:29:19.197197 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:19.197097    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:29:29.196775 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:29.196722    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:29:29.197339 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:29.197189    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:29:39.197731 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:39.197696    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:29:39.198229 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:39.198031    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:29:53.825208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.825167    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:53.858761 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.858727    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"]
Sep 11 16:29:53.859065 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.859015    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" containerID="cri-o://4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26" gracePeriod=30
Sep 11 16:29:53.859181 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.859066    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy" containerID="cri-o://7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa" gracePeriod=30
Sep 11 16:29:53.859181 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.859062    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" containerID="cri-o://04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b" gracePeriod=30
Sep 11 16:29:53.912750 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.912727    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:53.921726 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.921699    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"]
Sep 11 16:29:53.925090 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.925075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:53.927427 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.927408    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 11 16:29:53.927594 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.927576    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 11 16:29:53.936472 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:53.936451    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"]
Sep 11 16:29:54.015924 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.015886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:54.045721 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.045684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.045912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.045753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06bb9679-5d35-4c52-a955-a0bbbce530c0-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.045912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.045860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.046127 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.045911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06bb9679-5d35-4c52-a955-a0bbbce530c0-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.046127 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.045961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbxk\" (UniqueName: \"kubernetes.io/projected/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-api-access-4bbxk\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.147393 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.147310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06bb9679-5d35-4c52-a955-a0bbbce530c0-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.147393 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.147355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.147592 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.147480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06bb9679-5d35-4c52-a955-a0bbbce530c0-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.147592 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.147513    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbxk\" (UniqueName: \"kubernetes.io/projected/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-api-access-4bbxk\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.147592 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.147546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.147735 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.147693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06bb9679-5d35-4c52-a955-a0bbbce530c0-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.148166 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.148145    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.148249 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.148166    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.149986 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.149967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06bb9679-5d35-4c52-a955-a0bbbce530c0-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.157743 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.157719    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbxk\" (UniqueName: \"kubernetes.io/projected/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-api-access-4bbxk\") pod \"isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.237323 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.237287    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:54.512318 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.512267    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"]
Sep 11 16:29:54.514332 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:29:54.514306    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bb9679_5d35_4c52_a955_a0bbbce530c0.slice/crio-321c183fc881aae3491dbea433246fcb9558df45d27fa5667d1ff3d02137338e WatchSource:0}: Error finding container 321c183fc881aae3491dbea433246fcb9558df45d27fa5667d1ff3d02137338e: Status 404 returned error can't find the container with id 321c183fc881aae3491dbea433246fcb9558df45d27fa5667d1ff3d02137338e
Sep 11 16:29:54.519918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.519896    2576 generic.go:356] "Generic (PLEG): container finished" podID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerID="7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa" exitCode=2
Sep 11 16:29:54.520028 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:54.519971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerDied","Data":"7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa"}
Sep 11 16:29:55.524845 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:55.524808    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerStarted","Data":"aad5d29eab416da614e8c5ef12c09a7e32655daa1ff7dbbb57f0e35d6124d9f2"}
Sep 11 16:29:55.525362 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:55.524886    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerStarted","Data":"321c183fc881aae3491dbea433246fcb9558df45d27fa5667d1ff3d02137338e"}
Sep 11 16:29:58.193243 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:58.193196    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 11 16:29:58.538869 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:58.538841    2576 generic.go:356] "Generic (PLEG): container finished" podID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerID="aad5d29eab416da614e8c5ef12c09a7e32655daa1ff7dbbb57f0e35d6124d9f2" exitCode=0
Sep 11 16:29:58.538987 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:58.538910    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerDied","Data":"aad5d29eab416da614e8c5ef12c09a7e32655daa1ff7dbbb57f0e35d6124d9f2"}
Sep 11 16:29:59.196602 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.196552    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:29:59.197051 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.196884    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:29:59.545544 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.545456    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerStarted","Data":"84a18b222c5558654649c6c47d602a0f3f0e0145f61d7ae9527b771cdff92d38"}
Sep 11 16:29:59.545544 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.545526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerStarted","Data":"fbc040a1066a6d4eac71157baef021967ae674ca2d5d38ad2a50b2768c57cf16"}
Sep 11 16:29:59.545764 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.545564    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerStarted","Data":"c654e2825af9a87cc1a8eefcc39dcb8e30e2c91c7d5ed4ce3e5c4df30bfa4a58"}
Sep 11 16:29:59.545974 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.545927    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:59.546119 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.546012    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:29:59.547728 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.547686    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:29:59.548254 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.548231    2576 generic.go:356] "Generic (PLEG): container finished" podID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerID="4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26" exitCode=0
Sep 11 16:29:59.548319 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.548273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerDied","Data":"4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26"}
Sep 11 16:29:59.570055 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:29:59.570011    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podStartSLOduration=2.722455463 podStartE2EDuration="6.569997426s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.847541963s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:59.56846697 +0000 UTC m=+793.478268133" watchObservedRunningTime="2026-09-11 16:29:59.569997426 +0000 UTC m=+793.479798587"
Sep 11 16:30:00.552350 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:00.552319    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:30:00.552925 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:00.552453    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:30:00.553463 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:00.553438    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:01.556275 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:01.556230    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:30:01.556705 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:01.556532    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:03.193260 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:03.193220    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 11 16:30:06.561284 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:06.561254    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:30:06.561727 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:06.561697    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:30:06.562007 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:06.561977    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:08.043352 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:08.043264    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:08.192624 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:08.192585    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 11 16:30:08.192812 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:08.192730    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:30:09.196544 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:09.196501    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:30:09.197020 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:09.196843    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:13.181362 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:13.181327    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:13.192526 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:13.192487    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 11 16:30:13.349547 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:13.349502    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:16.562095 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:16.562049    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:30:16.562540 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:16.562517    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:18.192752 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:18.192709    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 11 16:30:19.197188 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:19.197132    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 11 16:30:19.197611 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:19.197293    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:30:19.197611 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:19.197432    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:19.197611 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:19.197517    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:30:20.301170 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:20.301120    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:20.362772 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:20.362733    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:23.193386 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:23.193338    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 11 16:30:24.010377 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.010352    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:30:24.192561 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.192534    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-proxy-tls\") pod \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") "
Sep 11 16:30:24.192723 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.192598    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-rbac-proxy-sar-config\") pod \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") "
Sep 11 16:30:24.192723 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.192618    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-cabundle-cert\") pod \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") "
Sep 11 16:30:24.192723 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.192645    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w7x7\" (UniqueName: \"kubernetes.io/projected/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-api-access-2w7x7\") pod \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") "
Sep 11 16:30:24.192723 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.192674    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kserve-provision-location\") pod \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\" (UID: \"5b5a4ad6-f121-40d3-882a-466e3a23bb4b\") "
Sep 11 16:30:24.193082 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.193052    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b5a4ad6-f121-40d3-882a-466e3a23bb4b" (UID: "5b5a4ad6-f121-40d3-882a-466e3a23bb4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:30:24.193082 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.193065    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b5a4ad6-f121-40d3-882a-466e3a23bb4b" (UID: "5b5a4ad6-f121-40d3-882a-466e3a23bb4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:30:24.193177 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.193076    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b5a4ad6-f121-40d3-882a-466e3a23bb4b" (UID: "5b5a4ad6-f121-40d3-882a-466e3a23bb4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:30:24.194971 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.194924    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b5a4ad6-f121-40d3-882a-466e3a23bb4b" (UID: "5b5a4ad6-f121-40d3-882a-466e3a23bb4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:30:24.195288 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.195006    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-api-access-2w7x7" (OuterVolumeSpecName: "kube-api-access-2w7x7") pod "5b5a4ad6-f121-40d3-882a-466e3a23bb4b" (UID: "5b5a4ad6-f121-40d3-882a-466e3a23bb4b"). InnerVolumeSpecName "kube-api-access-2w7x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:30:24.293757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.293710    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:30:24.293757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.293755    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:30:24.293998 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.293771    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w7x7\" (UniqueName: \"kubernetes.io/projected/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kube-api-access-2w7x7\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:30:24.293998 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.293785    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:30:24.293998 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.293799    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5a4ad6-f121-40d3-882a-466e3a23bb4b-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:30:24.643306 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.643272    2576 generic.go:356] "Generic (PLEG): container finished" podID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerID="04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b" exitCode=0
Sep 11 16:30:24.643451 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.643327    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerDied","Data":"04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b"}
Sep 11 16:30:24.643451 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.643330    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"
Sep 11 16:30:24.643451 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.643380    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r" event={"ID":"5b5a4ad6-f121-40d3-882a-466e3a23bb4b","Type":"ContainerDied","Data":"5c9212a67b41619675e9afacbb11f4adfb6233622a309cbb9173562b85117093"}
Sep 11 16:30:24.643451 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.643410    2576 scope.go:117] "RemoveContainer" containerID="04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b"
Sep 11 16:30:24.651471 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.651441    2576 scope.go:117] "RemoveContainer" containerID="7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa"
Sep 11 16:30:24.658863 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.658845    2576 scope.go:117] "RemoveContainer" containerID="4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26"
Sep 11 16:30:24.666180 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.666134    2576 scope.go:117] "RemoveContainer" containerID="2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46"
Sep 11 16:30:24.668341 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.668322    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"]
Sep 11 16:30:24.673864 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.673840    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r"]
Sep 11 16:30:24.674572 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.674556    2576 scope.go:117] "RemoveContainer" containerID="04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b"
Sep 11 16:30:24.674815 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:30:24.674795    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b\": container with ID starting with 04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b not found: ID does not exist" containerID="04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b"
Sep 11 16:30:24.674882 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.674829    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b"} err="failed to get container status \"04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b\": rpc error: code = NotFound desc = could not find container \"04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b\": container with ID starting with 04a53048c60493defe60e6f3227d4a91b5b6e8d820bf2e4c81519da03c3b181b not found: ID does not exist"
Sep 11 16:30:24.674882 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.674854    2576 scope.go:117] "RemoveContainer" containerID="7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa"
Sep 11 16:30:24.675183 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:30:24.675165    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa\": container with ID starting with 7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa not found: ID does not exist" containerID="7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa"
Sep 11 16:30:24.675224 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.675189    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa"} err="failed to get container status \"7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa\": rpc error: code = NotFound desc = could not find container \"7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa\": container with ID starting with 7efaf0e85329e0a4147fee97154f5cd8e62af6109876e23be3485bc255e719fa not found: ID does not exist"
Sep 11 16:30:24.675224 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.675204    2576 scope.go:117] "RemoveContainer" containerID="4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26"
Sep 11 16:30:24.675437 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:30:24.675418    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26\": container with ID starting with 4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26 not found: ID does not exist" containerID="4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26"
Sep 11 16:30:24.675514 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.675443    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26"} err="failed to get container status \"4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26\": rpc error: code = NotFound desc = could not find container \"4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26\": container with ID starting with 4cb6a599d7e8c7fe5bc2dd42be89bdf93e934091556a91bac0b9c292f2d1db26 not found: ID does not exist"
Sep 11 16:30:24.675514 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.675458    2576 scope.go:117] "RemoveContainer" containerID="2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46"
Sep 11 16:30:24.675695 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:30:24.675679    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46\": container with ID starting with 2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46 not found: ID does not exist" containerID="2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46"
Sep 11 16:30:24.675732 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:24.675702    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46"} err="failed to get container status \"2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46\": rpc error: code = NotFound desc = could not find container \"2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46\": container with ID starting with 2e727fa6121b560427a17cf4f6dfccaa5b79bb9c8be08b3922893e587b355f46 not found: ID does not exist"
Sep 11 16:30:26.562228 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:26.562190    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:30:26.562666 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:26.562643    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:26.603520 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:26.603491    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" path="/var/lib/kubelet/pods/5b5a4ad6-f121-40d3-882a-466e3a23bb4b/volumes"
Sep 11 16:30:36.561836 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:36.561791    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:30:36.562413 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:36.562248    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:46.562707 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:46.562665    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:30:46.563080 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:46.563010    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:56.561805 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:56.561759    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:30:56.562236 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:30:56.562212    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:06.562534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:06.562495    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:31:06.562928 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:06.562901    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:16.562116 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:16.562085    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:31:16.563189 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:16.563072    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:31:28.253747 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:28.253706    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:28.971568 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:28.971538    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"]
Sep 11 16:31:28.971905 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:28.971867    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy" containerID="cri-o://fbc040a1066a6d4eac71157baef021967ae674ca2d5d38ad2a50b2768c57cf16" gracePeriod=30
Sep 11 16:31:28.971905 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:28.971900    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" containerID="cri-o://84a18b222c5558654649c6c47d602a0f3f0e0145f61d7ae9527b771cdff92d38" gracePeriod=30
Sep 11 16:31:28.972106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:28.971867    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" containerID="cri-o://c654e2825af9a87cc1a8eefcc39dcb8e30e2c91c7d5ed4ce3e5c4df30bfa4a58" gracePeriod=30
Sep 11 16:31:28.981532 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:28.981507    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:29.086877 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.086849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:29.108976 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.108948    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"]
Sep 11 16:31:29.109244 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109233    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="storage-initializer"
Sep 11 16:31:29.109287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109246    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="storage-initializer"
Sep 11 16:31:29.109287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109255    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container"
Sep 11 16:31:29.109287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109261    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container"
Sep 11 16:31:29.109287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109268    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy"
Sep 11 16:31:29.109287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109273    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy"
Sep 11 16:31:29.109287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109280    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent"
Sep 11 16:31:29.109287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109285    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent"
Sep 11 16:31:29.109589 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109336    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kserve-container"
Sep 11 16:31:29.109589 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109342    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="agent"
Sep 11 16:31:29.109589 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.109348    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5a4ad6-f121-40d3-882a-466e3a23bb4b" containerName="kube-rbac-proxy"
Sep 11 16:31:29.112313 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.112295    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.116389 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.116371    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 11 16:31:29.116495 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.116370    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 11 16:31:29.130432 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.130410    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"]
Sep 11 16:31:29.186424 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.186391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bf32387-8421-4bc4-ba80-3318277aaea2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kr9lt\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.186585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.186431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2fd\" (UniqueName: \"kubernetes.io/projected/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-api-access-kt2fd\") pod \"message-dumper-predictor-6575bcfdfd-kr9lt\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.186585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.186455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-kr9lt\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.214835 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.214796    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:29.287399 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.287322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bf32387-8421-4bc4-ba80-3318277aaea2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kr9lt\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.287399 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.287366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2fd\" (UniqueName: \"kubernetes.io/projected/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-api-access-kt2fd\") pod \"message-dumper-predictor-6575bcfdfd-kr9lt\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.287879 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.287494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-kr9lt\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.288354 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.288326    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-kr9lt\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.290029 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.290001    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bf32387-8421-4bc4-ba80-3318277aaea2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kr9lt\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.299787 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.299764    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2fd\" (UniqueName: \"kubernetes.io/projected/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-api-access-kt2fd\") pod \"message-dumper-predictor-6575bcfdfd-kr9lt\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.422791 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.422757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:29.694042 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.694016    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"]
Sep 11 16:31:29.696906 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:31:29.696878    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf32387_8421_4bc4_ba80_3318277aaea2.slice/crio-543c0dc69fed88c32fff39160ea72d6bc31d3a62d7a9afb8d292edec22a89446 WatchSource:0}: Error finding container 543c0dc69fed88c32fff39160ea72d6bc31d3a62d7a9afb8d292edec22a89446: Status 404 returned error can't find the container with id 543c0dc69fed88c32fff39160ea72d6bc31d3a62d7a9afb8d292edec22a89446
Sep 11 16:31:29.698918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.698900    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:31:29.887267 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.887236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt" event={"ID":"4bf32387-8421-4bc4-ba80-3318277aaea2","Type":"ContainerStarted","Data":"543c0dc69fed88c32fff39160ea72d6bc31d3a62d7a9afb8d292edec22a89446"}
Sep 11 16:31:29.889742 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.889719    2576 generic.go:356] "Generic (PLEG): container finished" podID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerID="fbc040a1066a6d4eac71157baef021967ae674ca2d5d38ad2a50b2768c57cf16" exitCode=2
Sep 11 16:31:29.889862 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:29.889798    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerDied","Data":"fbc040a1066a6d4eac71157baef021967ae674ca2d5d38ad2a50b2768c57cf16"}
Sep 11 16:31:31.556891 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:31.556854    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 11 16:31:31.901510 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:31.901419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt" event={"ID":"4bf32387-8421-4bc4-ba80-3318277aaea2","Type":"ContainerStarted","Data":"243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e"}
Sep 11 16:31:31.901510 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:31.901483    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt" event={"ID":"4bf32387-8421-4bc4-ba80-3318277aaea2","Type":"ContainerStarted","Data":"985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26"}
Sep 11 16:31:31.901763 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:31.901517    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:31.901763 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:31.901637    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:31.903155 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:31.903131    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:31.923191 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:31.923141    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt" podStartSLOduration=1.522296378 podStartE2EDuration="2.923128513s" totalImagesPullingTime="1.400832135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:31:31.921720054 +0000 UTC m=+885.831521219" watchObservedRunningTime="2026-09-11 16:31:31.923128513 +0000 UTC m=+885.832929728"
Sep 11 16:31:33.911191 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:33.911097    2576 generic.go:356] "Generic (PLEG): container finished" podID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerID="c654e2825af9a87cc1a8eefcc39dcb8e30e2c91c7d5ed4ce3e5c4df30bfa4a58" exitCode=0
Sep 11 16:31:33.911191 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:33.911139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerDied","Data":"c654e2825af9a87cc1a8eefcc39dcb8e30e2c91c7d5ed4ce3e5c4df30bfa4a58"}
Sep 11 16:31:36.556764 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:36.556721    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 11 16:31:36.562116 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:36.562087    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:31:36.563697 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:36.563670    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:38.917884 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:38.917808    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:31:41.556699 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:41.556657    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 11 16:31:41.557142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:41.556798    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:31:43.339796 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:43.339764    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:46.505735 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:46.505702    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:31:46.506182 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:46.505815    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:31:46.511909 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:46.511890    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:31:46.512212 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:46.512196    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:31:46.556697 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:46.556662    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 11 16:31:46.561994 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:46.561968    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:31:46.563488 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:46.563468    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:48.471329 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.471284    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:48.512826 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.512793    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"]
Sep 11 16:31:48.516126 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.516110    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.521749 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.521714    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 11 16:31:48.521890 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.521726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 11 16:31:48.529755 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.529732    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"]
Sep 11 16:31:48.584843 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.584811    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:48.636705 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.636673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98t8g\" (UniqueName: \"kubernetes.io/projected/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-api-access-98t8g\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.636705 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.636705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.636898 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.636736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deeb805b-ce24-42c7-9f11-a9110dcc282a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.636898 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.636818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.636898 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.636857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deeb805b-ce24-42c7-9f11-a9110dcc282a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.737762 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.737681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deeb805b-ce24-42c7-9f11-a9110dcc282a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.737762 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.737729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.737967 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.737907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deeb805b-ce24-42c7-9f11-a9110dcc282a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.738025 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.737984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98t8g\" (UniqueName: \"kubernetes.io/projected/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-api-access-98t8g\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.738025 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.738013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.738135 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.738116    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deeb805b-ce24-42c7-9f11-a9110dcc282a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.738503 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.738484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.738572 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.738556    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.740776 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.740751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deeb805b-ce24-42c7-9f11-a9110dcc282a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.751551 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.751530    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98t8g\" (UniqueName: \"kubernetes.io/projected/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-api-access-98t8g\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:48.826463 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:48.826433    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:31:49.113734 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:31:49.113702    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeb805b_ce24_42c7_9f11_a9110dcc282a.slice/crio-85bb26a23fc64b88c1c9c2e28e0301da6e923bdd8af649de61d38bde3d9bc90c WatchSource:0}: Error finding container 85bb26a23fc64b88c1c9c2e28e0301da6e923bdd8af649de61d38bde3d9bc90c: Status 404 returned error can't find the container with id 85bb26a23fc64b88c1c9c2e28e0301da6e923bdd8af649de61d38bde3d9bc90c
Sep 11 16:31:49.114668 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.114636    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"]
Sep 11 16:31:49.146050 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.146015    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:49.241838 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.241799    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"]
Sep 11 16:31:49.245532 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.245509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.248339 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.248310    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 11 16:31:49.248466 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.248373    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 11 16:31:49.267131 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.267106    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"]
Sep 11 16:31:49.343361 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.343332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kserve-provision-location\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.343361 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.343366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.343568 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.343403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lhh9\" (UniqueName: \"kubernetes.io/projected/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-api-access-4lhh9\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.343568 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.343467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-proxy-tls\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.343568 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.343495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-cabundle-cert\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.443975 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.443917    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhh9\" (UniqueName: \"kubernetes.io/projected/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-api-access-4lhh9\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.444156 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.443994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-proxy-tls\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.444156 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.444039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-cabundle-cert\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.444156 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.444091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kserve-provision-location\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.444156 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.444127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.444638 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.444613    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kserve-provision-location\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.444727 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.444674    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-cabundle-cert\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.444789 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.444769    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.446648 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.446627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-proxy-tls\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.455337 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.455311    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhh9\" (UniqueName: \"kubernetes.io/projected/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-api-access-4lhh9\") pod \"isvc-logger-predictor-6bdfd88f5f-rtc26\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.467611 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.467583    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:49.555310 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.555274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:49.838463 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.838386    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"]
Sep 11 16:31:49.840720 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:31:49.840689    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda782a8f4_ba76_4486_b9f8_3c9f28d6a5c0.slice/crio-27228d3dc8af1adbdbf7ac686361497860c525e5730a1188cf4ea81b0c90dcfa WatchSource:0}: Error finding container 27228d3dc8af1adbdbf7ac686361497860c525e5730a1188cf4ea81b0c90dcfa: Status 404 returned error can't find the container with id 27228d3dc8af1adbdbf7ac686361497860c525e5730a1188cf4ea81b0c90dcfa
Sep 11 16:31:49.974060 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.974024    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerStarted","Data":"d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e"}
Sep 11 16:31:49.974262 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.974098    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerStarted","Data":"27228d3dc8af1adbdbf7ac686361497860c525e5730a1188cf4ea81b0c90dcfa"}
Sep 11 16:31:49.975802 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.975772    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" event={"ID":"deeb805b-ce24-42c7-9f11-a9110dcc282a","Type":"ContainerStarted","Data":"9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565"}
Sep 11 16:31:49.975912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:49.975821    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" event={"ID":"deeb805b-ce24-42c7-9f11-a9110dcc282a","Type":"ContainerStarted","Data":"85bb26a23fc64b88c1c9c2e28e0301da6e923bdd8af649de61d38bde3d9bc90c"}
Sep 11 16:31:51.557148 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:51.557105    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 11 16:31:53.749194 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:53.749157    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:53.998209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:53.998178    2576 generic.go:356] "Generic (PLEG): container finished" podID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerID="9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565" exitCode=0
Sep 11 16:31:53.998375 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:53.998252    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" event={"ID":"deeb805b-ce24-42c7-9f11-a9110dcc282a","Type":"ContainerDied","Data":"9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565"}
Sep 11 16:31:53.999878 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:53.999853    2576 generic.go:356] "Generic (PLEG): container finished" podID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerID="d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e" exitCode=0
Sep 11 16:31:54.000018 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:53.999927    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerDied","Data":"d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e"}
Sep 11 16:31:55.009676 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:55.009519    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerStarted","Data":"2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c"}
Sep 11 16:31:55.009676 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:55.009602    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerStarted","Data":"87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0"}
Sep 11 16:31:55.009676 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:55.009638    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerStarted","Data":"076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139"}
Sep 11 16:31:55.011568 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:55.011542    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:55.011694 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:55.011613    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:55.011694 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:55.011650    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:31:55.013282 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:55.013246    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:55.013452 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:55.013382    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:31:55.039549 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:55.039491    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podStartSLOduration=2.293131917 podStartE2EDuration="6.039469556s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.746337639s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:31:55.037996056 +0000 UTC m=+908.947797221" watchObservedRunningTime="2026-09-11 16:31:55.039469556 +0000 UTC m=+908.949270719"
Sep 11 16:31:56.020431 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:56.020196    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:31:56.021417 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:56.021351    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:56.557510 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:56.557270    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 11 16:31:56.562097 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:56.562049    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Sep 11 16:31:56.562255 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:56.562192    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:31:56.563065 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:56.562964    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:56.563238 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:31:56.563079    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:32:00.045132 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:00.045093    2576 generic.go:356] "Generic (PLEG): container finished" podID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerID="84a18b222c5558654649c6c47d602a0f3f0e0145f61d7ae9527b771cdff92d38" exitCode=0
Sep 11 16:32:00.045615 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:00.045188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerDied","Data":"84a18b222c5558654649c6c47d602a0f3f0e0145f61d7ae9527b771cdff92d38"}
Sep 11 16:32:01.024584 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:01.024556    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:32:01.025618 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:01.025407    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:32:01.025843 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:01.025805    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:02.557230 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:02.557186    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 11 16:32:03.295502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.295412    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:32:03.364799 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.364760    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06bb9679-5d35-4c52-a955-a0bbbce530c0-kserve-provision-location\") pod \"06bb9679-5d35-4c52-a955-a0bbbce530c0\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") "
Sep 11 16:32:03.364980 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.364843    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bbxk\" (UniqueName: \"kubernetes.io/projected/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-api-access-4bbxk\") pod \"06bb9679-5d35-4c52-a955-a0bbbce530c0\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") "
Sep 11 16:32:03.364980 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.364888    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-rbac-proxy-sar-config\") pod \"06bb9679-5d35-4c52-a955-a0bbbce530c0\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") "
Sep 11 16:32:03.364980 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.364918    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-cabundle-cert\") pod \"06bb9679-5d35-4c52-a955-a0bbbce530c0\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") "
Sep 11 16:32:03.365090 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.365014    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06bb9679-5d35-4c52-a955-a0bbbce530c0-proxy-tls\") pod \"06bb9679-5d35-4c52-a955-a0bbbce530c0\" (UID: \"06bb9679-5d35-4c52-a955-a0bbbce530c0\") "
Sep 11 16:32:03.365125 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.365106    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bb9679-5d35-4c52-a955-a0bbbce530c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06bb9679-5d35-4c52-a955-a0bbbce530c0" (UID: "06bb9679-5d35-4c52-a955-a0bbbce530c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:03.365253 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.365226    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06bb9679-5d35-4c52-a955-a0bbbce530c0" (UID: "06bb9679-5d35-4c52-a955-a0bbbce530c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:32:03.365325 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.365272    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06bb9679-5d35-4c52-a955-a0bbbce530c0" (UID: "06bb9679-5d35-4c52-a955-a0bbbce530c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:32:03.365325 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.365302    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06bb9679-5d35-4c52-a955-a0bbbce530c0-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:32:03.365424 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.365324    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:32:03.367841 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.367799    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-api-access-4bbxk" (OuterVolumeSpecName: "kube-api-access-4bbxk") pod "06bb9679-5d35-4c52-a955-a0bbbce530c0" (UID: "06bb9679-5d35-4c52-a955-a0bbbce530c0"). InnerVolumeSpecName "kube-api-access-4bbxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:32:03.368900 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.368867    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bb9679-5d35-4c52-a955-a0bbbce530c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06bb9679-5d35-4c52-a955-a0bbbce530c0" (UID: "06bb9679-5d35-4c52-a955-a0bbbce530c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:32:03.466555 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.466520    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06bb9679-5d35-4c52-a955-a0bbbce530c0-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:32:03.466555 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.466556    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06bb9679-5d35-4c52-a955-a0bbbce530c0-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:32:03.466555 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:03.466567    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbxk\" (UniqueName: \"kubernetes.io/projected/06bb9679-5d35-4c52-a955-a0bbbce530c0-kube-api-access-4bbxk\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:32:04.068894 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:04.068851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r" event={"ID":"06bb9679-5d35-4c52-a955-a0bbbce530c0","Type":"ContainerDied","Data":"321c183fc881aae3491dbea433246fcb9558df45d27fa5667d1ff3d02137338e"}
Sep 11 16:32:04.069343 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:04.068868    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"
Sep 11 16:32:04.069343 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:04.069014    2576 scope.go:117] "RemoveContainer" containerID="84a18b222c5558654649c6c47d602a0f3f0e0145f61d7ae9527b771cdff92d38"
Sep 11 16:32:04.100410 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:04.100348    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"]
Sep 11 16:32:04.102815 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:04.102788    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r"]
Sep 11 16:32:04.605247 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:04.605182    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" path="/var/lib/kubelet/pods/06bb9679-5d35-4c52-a955-a0bbbce530c0/volumes"
Sep 11 16:32:04.927496 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:04.927292    2576 scope.go:117] "RemoveContainer" containerID="fbc040a1066a6d4eac71157baef021967ae674ca2d5d38ad2a50b2768c57cf16"
Sep 11 16:32:04.936163 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:04.936130    2576 scope.go:117] "RemoveContainer" containerID="c654e2825af9a87cc1a8eefcc39dcb8e30e2c91c7d5ed4ce3e5c4df30bfa4a58"
Sep 11 16:32:04.944825 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:04.944770    2576 scope.go:117] "RemoveContainer" containerID="aad5d29eab416da614e8c5ef12c09a7e32655daa1ff7dbbb57f0e35d6124d9f2"
Sep 11 16:32:11.025908 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:11.025855    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:32:11.026470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:11.026361    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:18.137380 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:18.137342    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" event={"ID":"deeb805b-ce24-42c7-9f11-a9110dcc282a","Type":"ContainerStarted","Data":"afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354"}
Sep 11 16:32:18.137800 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:18.137432    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" event={"ID":"deeb805b-ce24-42c7-9f11-a9110dcc282a","Type":"ContainerStarted","Data":"cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae"}
Sep 11 16:32:18.137800 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:18.137708    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:32:18.137887 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:18.137869    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:32:18.139168 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:18.139130    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 11 16:32:18.159974 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:18.159909    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podStartSLOduration=3.023474776 podStartE2EDuration="30.159896303s" totalImagesPullingTime="23.078296297s" totalInitContainerRuntime="4.05812523s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:32:18.158446427 +0000 UTC m=+932.068247589" watchObservedRunningTime="2026-09-11 16:32:18.159896303 +0000 UTC m=+932.069697524"
Sep 11 16:32:19.141529 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:19.141493    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 11 16:32:21.025650 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:21.025603    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:32:21.026114 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:21.026092    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:24.146505 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:24.146473    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:32:24.147020 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:24.146991    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 11 16:32:31.025470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:31.025426    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:32:31.025872 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:31.025735    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:34.147636 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:34.147599    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 11 16:32:41.025621 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:41.025574    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:32:41.026115 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:41.026032    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:44.147561 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:44.147522    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 11 16:32:51.025433 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:51.025397    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:32:51.025869 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:51.025667    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:54.147145 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:32:54.147095    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 11 16:33:01.025707 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:01.025663    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:33:01.026142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:01.026079    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:33:02.598359 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:02.598313    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:33:02.598832 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:02.598593    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:33:04.147634 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:04.147599    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 11 16:33:12.603404 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:12.603337    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:33:12.603752 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:12.603420    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:33:12.970597 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:12.970558    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:14.147278 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:14.147242    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 11 16:33:14.220422 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:14.220389    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:24.118290 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.118260    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-kr9lt_4bf32387-8421-4bc4-ba80-3318277aaea2/kserve-container/0.log"
Sep 11 16:33:24.147015 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.146976    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 11 16:33:24.364833 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.364795    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"]
Sep 11 16:33:24.365253 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.365152    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" containerID="cri-o://076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139" gracePeriod=30
Sep 11 16:33:24.365253 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.365195    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" containerID="cri-o://2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c" gracePeriod=30
Sep 11 16:33:24.365438 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.365208    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy" containerID="cri-o://87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0" gracePeriod=30
Sep 11 16:33:24.374727 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.374689    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:24.487728 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.487697    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:24.533071 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.533042    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"]
Sep 11 16:33:24.533346 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.533312    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt" podUID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerName="kserve-container" containerID="cri-o://985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26" gracePeriod=30
Sep 11 16:33:24.533486 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.533351    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt" podUID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerName="kube-rbac-proxy" containerID="cri-o://243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e" gracePeriod=30
Sep 11 16:33:24.574450 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.574413    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:24.779161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.779129    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:33:24.877006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.876978    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bf32387-8421-4bc4-ba80-3318277aaea2-proxy-tls\") pod \"4bf32387-8421-4bc4-ba80-3318277aaea2\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") "
Sep 11 16:33:24.877006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.877009    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt2fd\" (UniqueName: \"kubernetes.io/projected/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-api-access-kt2fd\") pod \"4bf32387-8421-4bc4-ba80-3318277aaea2\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") "
Sep 11 16:33:24.877239 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.877044    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-rbac-proxy-sar-config\") pod \"4bf32387-8421-4bc4-ba80-3318277aaea2\" (UID: \"4bf32387-8421-4bc4-ba80-3318277aaea2\") "
Sep 11 16:33:24.877456 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.877430    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4bf32387-8421-4bc4-ba80-3318277aaea2" (UID: "4bf32387-8421-4bc4-ba80-3318277aaea2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:24.879268 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.879242    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-api-access-kt2fd" (OuterVolumeSpecName: "kube-api-access-kt2fd") pod "4bf32387-8421-4bc4-ba80-3318277aaea2" (UID: "4bf32387-8421-4bc4-ba80-3318277aaea2"). InnerVolumeSpecName "kube-api-access-kt2fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:24.879268 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.879259    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf32387-8421-4bc4-ba80-3318277aaea2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bf32387-8421-4bc4-ba80-3318277aaea2" (UID: "4bf32387-8421-4bc4-ba80-3318277aaea2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:24.978017 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.977995    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bf32387-8421-4bc4-ba80-3318277aaea2-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:24.978017 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.978017    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2fd\" (UniqueName: \"kubernetes.io/projected/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-api-access-kt2fd\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:24.978153 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:24.978026    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bf32387-8421-4bc4-ba80-3318277aaea2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:25.406380 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.406349    2576 generic.go:356] "Generic (PLEG): container finished" podID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerID="243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e" exitCode=2
Sep 11 16:33:25.406380 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.406371    2576 generic.go:356] "Generic (PLEG): container finished" podID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerID="985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26" exitCode=2
Sep 11 16:33:25.406839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.406400    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"
Sep 11 16:33:25.406839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.406437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt" event={"ID":"4bf32387-8421-4bc4-ba80-3318277aaea2","Type":"ContainerDied","Data":"243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e"}
Sep 11 16:33:25.406839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.406500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt" event={"ID":"4bf32387-8421-4bc4-ba80-3318277aaea2","Type":"ContainerDied","Data":"985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26"}
Sep 11 16:33:25.406839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.406527    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt" event={"ID":"4bf32387-8421-4bc4-ba80-3318277aaea2","Type":"ContainerDied","Data":"543c0dc69fed88c32fff39160ea72d6bc31d3a62d7a9afb8d292edec22a89446"}
Sep 11 16:33:25.406839 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.406563    2576 scope.go:117] "RemoveContainer" containerID="243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e"
Sep 11 16:33:25.416238 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.416215    2576 generic.go:356] "Generic (PLEG): container finished" podID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerID="87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0" exitCode=2
Sep 11 16:33:25.416358 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.416261    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerDied","Data":"87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0"}
Sep 11 16:33:25.417974 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.417956    2576 scope.go:117] "RemoveContainer" containerID="985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26"
Sep 11 16:33:25.425094 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.425077    2576 scope.go:117] "RemoveContainer" containerID="243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e"
Sep 11 16:33:25.425337 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:25.425319    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e\": container with ID starting with 243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e not found: ID does not exist" containerID="243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e"
Sep 11 16:33:25.425387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.425344    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e"} err="failed to get container status \"243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e\": rpc error: code = NotFound desc = could not find container \"243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e\": container with ID starting with 243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e not found: ID does not exist"
Sep 11 16:33:25.425387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.425360    2576 scope.go:117] "RemoveContainer" containerID="985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26"
Sep 11 16:33:25.425623 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:25.425605    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26\": container with ID starting with 985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26 not found: ID does not exist" containerID="985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26"
Sep 11 16:33:25.425677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.425627    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26"} err="failed to get container status \"985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26\": rpc error: code = NotFound desc = could not find container \"985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26\": container with ID starting with 985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26 not found: ID does not exist"
Sep 11 16:33:25.425677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.425643    2576 scope.go:117] "RemoveContainer" containerID="243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e"
Sep 11 16:33:25.425852 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.425836    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e"} err="failed to get container status \"243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e\": rpc error: code = NotFound desc = could not find container \"243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e\": container with ID starting with 243469f26389b6257bd7b49620b2ed41a9fb15b3448cab2eceefa2e0b2c6af0e not found: ID does not exist"
Sep 11 16:33:25.425897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.425852    2576 scope.go:117] "RemoveContainer" containerID="985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26"
Sep 11 16:33:25.426108 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.426090    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26"} err="failed to get container status \"985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26\": rpc error: code = NotFound desc = could not find container \"985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26\": container with ID starting with 985026acb008a91e26445e595bcff61c11ffbbde598ca5f45147460546ff5d26 not found: ID does not exist"
Sep 11 16:33:25.432518 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.432498    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"]
Sep 11 16:33:25.436243 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:25.436221    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt"]
Sep 11 16:33:26.020072 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:26.020034    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 11 16:33:26.602533 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:26.602502    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bf32387-8421-4bc4-ba80-3318277aaea2" path="/var/lib/kubelet/pods/4bf32387-8421-4bc4-ba80-3318277aaea2/volumes"
Sep 11 16:33:29.435543 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:29.435511    2576 generic.go:356] "Generic (PLEG): container finished" podID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerID="076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139" exitCode=0
Sep 11 16:33:29.435967 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:29.435582    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerDied","Data":"076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139"}
Sep 11 16:33:31.019497 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:31.019452    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 11 16:33:32.599002 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:32.598955    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:33:32.599487 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:32.599249    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:33:34.147621 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:34.147589    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:33:36.020418 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:36.020370    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 11 16:33:36.020853 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:36.020508    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:33:38.606876 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:38.606846    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"]
Sep 11 16:33:38.607300 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:38.607068    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" containerID="cri-o://cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae" gracePeriod=30
Sep 11 16:33:38.607300 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:38.607092    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kube-rbac-proxy" containerID="cri-o://afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354" gracePeriod=30
Sep 11 16:33:38.633035 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:38.633009    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:39.142622 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:39.142580    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 11 16:33:39.482446 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:39.482414    2576 generic.go:356] "Generic (PLEG): container finished" podID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerID="afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354" exitCode=2
Sep 11 16:33:39.482607 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:39.482483    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" event={"ID":"deeb805b-ce24-42c7-9f11-a9110dcc282a","Type":"ContainerDied","Data":"afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354"}
Sep 11 16:33:41.020092 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:41.020050    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 11 16:33:42.598377 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:42.598333    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:33:42.598795 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:42.598511    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:33:43.781918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.781863    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:43.863280 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863247    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"]
Sep 11 16:33:43.863574 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863561    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="storage-initializer"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863577    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="storage-initializer"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863588    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863594    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863601    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863607    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863614    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863619    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863626    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerName="kserve-container"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863631    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerName="kserve-container"
Sep 11 16:33:43.863633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863637    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerName="kube-rbac-proxy"
Sep 11 16:33:43.863918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863642    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerName="kube-rbac-proxy"
Sep 11 16:33:43.863918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863690    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kserve-container"
Sep 11 16:33:43.863918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863696    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="kube-rbac-proxy"
Sep 11 16:33:43.863918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863705    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerName="kserve-container"
Sep 11 16:33:43.863918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863712    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf32387-8421-4bc4-ba80-3318277aaea2" containerName="kube-rbac-proxy"
Sep 11 16:33:43.863918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.863719    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="06bb9679-5d35-4c52-a955-a0bbbce530c0" containerName="agent"
Sep 11 16:33:43.868017 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.866902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:43.872703 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.872680    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 11 16:33:43.875976 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.875953    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 11 16:33:43.887908 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.887879    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"]
Sep 11 16:33:43.917473 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.917442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e91f23cf-5020-4b33-957d-333e4115671a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:43.917617 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.917498    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:43.917617 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.917524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e91f23cf-5020-4b33-957d-333e4115671a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:43.917617 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.917569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:43.917732 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:43.917630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmsf\" (UniqueName: \"kubernetes.io/projected/e91f23cf-5020-4b33-957d-333e4115671a-kube-api-access-cjmsf\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.018930 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.018898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.019111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.018962    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e91f23cf-5020-4b33-957d-333e4115671a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.019111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.018992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.019111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.019019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmsf\" (UniqueName: \"kubernetes.io/projected/e91f23cf-5020-4b33-957d-333e4115671a-kube-api-access-cjmsf\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.019111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.019047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e91f23cf-5020-4b33-957d-333e4115671a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.019316 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:44.019199    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 11 16:33:44.019316 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:44.019271    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e91f23cf-5020-4b33-957d-333e4115671a-proxy-tls podName:e91f23cf-5020-4b33-957d-333e4115671a nodeName:}" failed. No retries permitted until 2026-09-11 16:33:44.519250825 +0000 UTC m=+1018.429051965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e91f23cf-5020-4b33-957d-333e4115671a-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" (UID: "e91f23cf-5020-4b33-957d-333e4115671a") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 11 16:33:44.019651 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.019628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e91f23cf-5020-4b33-957d-333e4115671a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.019756 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.019734    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.019793 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.019735    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.029064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.029043    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmsf\" (UniqueName: \"kubernetes.io/projected/e91f23cf-5020-4b33-957d-333e4115671a-kube-api-access-cjmsf\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.116987 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.116954    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:44.155023 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.154996    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:33:44.221105 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.221072    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98t8g\" (UniqueName: \"kubernetes.io/projected/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-api-access-98t8g\") pod \"deeb805b-ce24-42c7-9f11-a9110dcc282a\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") "
Sep 11 16:33:44.221273 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.221127    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-cabundle-cert\") pod \"deeb805b-ce24-42c7-9f11-a9110dcc282a\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") "
Sep 11 16:33:44.221273 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.221182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deeb805b-ce24-42c7-9f11-a9110dcc282a-kserve-provision-location\") pod \"deeb805b-ce24-42c7-9f11-a9110dcc282a\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") "
Sep 11 16:33:44.221273 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.221202    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-rbac-proxy-sar-config\") pod \"deeb805b-ce24-42c7-9f11-a9110dcc282a\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") "
Sep 11 16:33:44.221273 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.221221    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deeb805b-ce24-42c7-9f11-a9110dcc282a-proxy-tls\") pod \"deeb805b-ce24-42c7-9f11-a9110dcc282a\" (UID: \"deeb805b-ce24-42c7-9f11-a9110dcc282a\") "
Sep 11 16:33:44.221558 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.221523    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeb805b-ce24-42c7-9f11-a9110dcc282a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "deeb805b-ce24-42c7-9f11-a9110dcc282a" (UID: "deeb805b-ce24-42c7-9f11-a9110dcc282a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:33:44.221558 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.221539    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "deeb805b-ce24-42c7-9f11-a9110dcc282a" (UID: "deeb805b-ce24-42c7-9f11-a9110dcc282a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:44.221558 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.221550    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "deeb805b-ce24-42c7-9f11-a9110dcc282a" (UID: "deeb805b-ce24-42c7-9f11-a9110dcc282a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:44.223459 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.223440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deeb805b-ce24-42c7-9f11-a9110dcc282a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "deeb805b-ce24-42c7-9f11-a9110dcc282a" (UID: "deeb805b-ce24-42c7-9f11-a9110dcc282a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:44.223558 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.223463    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-api-access-98t8g" (OuterVolumeSpecName: "kube-api-access-98t8g") pod "deeb805b-ce24-42c7-9f11-a9110dcc282a" (UID: "deeb805b-ce24-42c7-9f11-a9110dcc282a"). InnerVolumeSpecName "kube-api-access-98t8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:44.322130 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.322096    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:44.322130 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.322126    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/deeb805b-ce24-42c7-9f11-a9110dcc282a-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:44.322130 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.322137    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:44.322352 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.322146    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deeb805b-ce24-42c7-9f11-a9110dcc282a-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:44.322352 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.322155    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98t8g\" (UniqueName: \"kubernetes.io/projected/deeb805b-ce24-42c7-9f11-a9110dcc282a-kube-api-access-98t8g\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:44.503865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.503834    2576 generic.go:356] "Generic (PLEG): container finished" podID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerID="cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae" exitCode=0
Sep 11 16:33:44.504048 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.503958    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" event={"ID":"deeb805b-ce24-42c7-9f11-a9110dcc282a","Type":"ContainerDied","Data":"cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae"}
Sep 11 16:33:44.504048 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.503968    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"
Sep 11 16:33:44.504048 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.504008    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" event={"ID":"deeb805b-ce24-42c7-9f11-a9110dcc282a","Type":"ContainerDied","Data":"85bb26a23fc64b88c1c9c2e28e0301da6e923bdd8af649de61d38bde3d9bc90c"}
Sep 11 16:33:44.504048 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.504033    2576 scope.go:117] "RemoveContainer" containerID="afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354"
Sep 11 16:33:44.512922 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.512902    2576 scope.go:117] "RemoveContainer" containerID="cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae"
Sep 11 16:33:44.520287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.520267    2576 scope.go:117] "RemoveContainer" containerID="9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565"
Sep 11 16:33:44.524480 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.524458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e91f23cf-5020-4b33-957d-333e4115671a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.526706 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.526687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e91f23cf-5020-4b33-957d-333e4115671a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:44.527170 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.527153    2576 scope.go:117] "RemoveContainer" containerID="afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354"
Sep 11 16:33:44.527446 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:44.527425    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354\": container with ID starting with afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354 not found: ID does not exist" containerID="afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354"
Sep 11 16:33:44.527493 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.527459    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354"} err="failed to get container status \"afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354\": rpc error: code = NotFound desc = could not find container \"afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354\": container with ID starting with afc5e2fcff5c22f96812d8d6d74143bf4623b3e3de1e13ab0ef879c51c59d354 not found: ID does not exist"
Sep 11 16:33:44.527493 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.527476    2576 scope.go:117] "RemoveContainer" containerID="cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae"
Sep 11 16:33:44.527727 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:44.527705    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae\": container with ID starting with cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae not found: ID does not exist" containerID="cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae"
Sep 11 16:33:44.527849 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.527733    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae"} err="failed to get container status \"cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae\": rpc error: code = NotFound desc = could not find container \"cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae\": container with ID starting with cfb3b5b47efd55346d9955e50284f7e38375b5323e8561505b0e8206417787ae not found: ID does not exist"
Sep 11 16:33:44.527849 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.527748    2576 scope.go:117] "RemoveContainer" containerID="9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565"
Sep 11 16:33:44.528086 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:44.528036    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565\": container with ID starting with 9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565 not found: ID does not exist" containerID="9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565"
Sep 11 16:33:44.528086 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.528063    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565"} err="failed to get container status \"9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565\": rpc error: code = NotFound desc = could not find container \"9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565\": container with ID starting with 9ac97e54345be6cbdd2ffc3feac5ddda8f83aeed754cf04a470a7b2aee2ce565 not found: ID does not exist"
Sep 11 16:33:44.530194 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.530175    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"]
Sep 11 16:33:44.537462 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.537442    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw"]
Sep 11 16:33:44.602877 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.602846    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" path="/var/lib/kubelet/pods/deeb805b-ce24-42c7-9f11-a9110dcc282a/volumes"
Sep 11 16:33:44.779663 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:44.779593    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:45.053310 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:45.053288    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"]
Sep 11 16:33:45.055469 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:33:45.055443    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91f23cf_5020_4b33_957d_333e4115671a.slice/crio-a754188bb73aa673c118fe7561796c96638f0bb4237b15e92e874396529acbe0 WatchSource:0}: Error finding container a754188bb73aa673c118fe7561796c96638f0bb4237b15e92e874396529acbe0: Status 404 returned error can't find the container with id a754188bb73aa673c118fe7561796c96638f0bb4237b15e92e874396529acbe0
Sep 11 16:33:45.143087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:45.143045    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": context deadline exceeded"
Sep 11 16:33:45.147617 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:45.147586    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: i/o timeout"
Sep 11 16:33:45.509031 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:45.509000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" event={"ID":"e91f23cf-5020-4b33-957d-333e4115671a","Type":"ContainerStarted","Data":"2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf"}
Sep 11 16:33:45.509222 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:45.509066    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" event={"ID":"e91f23cf-5020-4b33-957d-333e4115671a","Type":"ContainerStarted","Data":"a754188bb73aa673c118fe7561796c96638f0bb4237b15e92e874396529acbe0"}
Sep 11 16:33:46.019550 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:46.019514    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 11 16:33:49.527537 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:49.527503    2576 generic.go:356] "Generic (PLEG): container finished" podID="e91f23cf-5020-4b33-957d-333e4115671a" containerID="2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf" exitCode=0
Sep 11 16:33:49.528005 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:49.527583    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" event={"ID":"e91f23cf-5020-4b33-957d-333e4115671a","Type":"ContainerDied","Data":"2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf"}
Sep 11 16:33:50.536205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:50.536174    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" event={"ID":"e91f23cf-5020-4b33-957d-333e4115671a","Type":"ContainerStarted","Data":"f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3"}
Sep 11 16:33:50.536625 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:50.536234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" event={"ID":"e91f23cf-5020-4b33-957d-333e4115671a","Type":"ContainerStarted","Data":"18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3"}
Sep 11 16:33:50.536625 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:50.536432    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:50.536625 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:50.536586    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:50.558522 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:50.558472    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" podStartSLOduration=3.6339556650000002 podStartE2EDuration="7.55845764s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.924501975s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:33:50.557528621 +0000 UTC m=+1024.467329783" watchObservedRunningTime="2026-09-11 16:33:50.55845764 +0000 UTC m=+1024.468258799"
Sep 11 16:33:51.020435 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:51.020398    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 11 16:33:52.598588 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:52.598529    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 11 16:33:52.599034 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:52.598857    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:33:52.603758 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:52.603739    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:33:52.603838 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:52.603825    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:33:54.513039 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.513016    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:33:54.555589 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.555556    2576 generic.go:356] "Generic (PLEG): container finished" podID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerID="2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c" exitCode=0
Sep 11 16:33:54.555757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.555615    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"
Sep 11 16:33:54.555757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.555623    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerDied","Data":"2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c"}
Sep 11 16:33:54.555757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.555692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26" event={"ID":"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0","Type":"ContainerDied","Data":"27228d3dc8af1adbdbf7ac686361497860c525e5730a1188cf4ea81b0c90dcfa"}
Sep 11 16:33:54.555757 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.555733    2576 scope.go:117] "RemoveContainer" containerID="2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c"
Sep 11 16:33:54.564148 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.564133    2576 scope.go:117] "RemoveContainer" containerID="87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0"
Sep 11 16:33:54.571561 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.571541    2576 scope.go:117] "RemoveContainer" containerID="076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139"
Sep 11 16:33:54.578961 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.578923    2576 scope.go:117] "RemoveContainer" containerID="d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e"
Sep 11 16:33:54.585848 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.585823    2576 scope.go:117] "RemoveContainer" containerID="2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c"
Sep 11 16:33:54.586157 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:54.586136    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c\": container with ID starting with 2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c not found: ID does not exist" containerID="2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c"
Sep 11 16:33:54.586205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.586168    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c"} err="failed to get container status \"2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c\": rpc error: code = NotFound desc = could not find container \"2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c\": container with ID starting with 2edf048a9c87b462db241b9c1cea151ba28f3f6205b8879588738e0d0199fd5c not found: ID does not exist"
Sep 11 16:33:54.586205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.586188    2576 scope.go:117] "RemoveContainer" containerID="87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0"
Sep 11 16:33:54.586466 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:54.586447    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0\": container with ID starting with 87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0 not found: ID does not exist" containerID="87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0"
Sep 11 16:33:54.586523 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.586471    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0"} err="failed to get container status \"87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0\": rpc error: code = NotFound desc = could not find container \"87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0\": container with ID starting with 87618dbc4743cf1ea2b5bc102d3eefca694b1310b4a777522a569dc6927631d0 not found: ID does not exist"
Sep 11 16:33:54.586523 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.586488    2576 scope.go:117] "RemoveContainer" containerID="076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139"
Sep 11 16:33:54.586712 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:54.586695    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139\": container with ID starting with 076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139 not found: ID does not exist" containerID="076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139"
Sep 11 16:33:54.586748 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.586716    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139"} err="failed to get container status \"076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139\": rpc error: code = NotFound desc = could not find container \"076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139\": container with ID starting with 076a42add7df3dbb0eb533d54d29b855139f170d993e6cc292e567b05f4ed139 not found: ID does not exist"
Sep 11 16:33:54.586748 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.586730    2576 scope.go:117] "RemoveContainer" containerID="d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e"
Sep 11 16:33:54.586954 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:33:54.586923    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e\": container with ID starting with d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e not found: ID does not exist" containerID="d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e"
Sep 11 16:33:54.587013 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.586957    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e"} err="failed to get container status \"d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e\": rpc error: code = NotFound desc = could not find container \"d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e\": container with ID starting with d13d1c90afb52791fc6846c3d6cd1be7773deb22422d363f807300774429515e not found: ID does not exist"
Sep 11 16:33:54.609041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.605196    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-rbac-proxy-sar-config\") pod \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") "
Sep 11 16:33:54.609041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.605232    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-cabundle-cert\") pod \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") "
Sep 11 16:33:54.609041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.605311    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lhh9\" (UniqueName: \"kubernetes.io/projected/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-api-access-4lhh9\") pod \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") "
Sep 11 16:33:54.609041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.605354    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-proxy-tls\") pod \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") "
Sep 11 16:33:54.609041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.605412    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kserve-provision-location\") pod \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\" (UID: \"a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0\") "
Sep 11 16:33:54.609041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.606121    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" (UID: "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:33:54.609041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.606186    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" (UID: "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:54.609041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.606195    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" (UID: "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:54.611451 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.611424    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" (UID: "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:54.611583 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.611453    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-api-access-4lhh9" (OuterVolumeSpecName: "kube-api-access-4lhh9") pod "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" (UID: "a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0"). InnerVolumeSpecName "kube-api-access-4lhh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:54.706124 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.706086    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lhh9\" (UniqueName: \"kubernetes.io/projected/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-api-access-4lhh9\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:54.706278 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.706127    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:54.706278 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.706141    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:54.706278 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.706153    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:54.706278 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.706166    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:33:54.880276 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.880245    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"]
Sep 11 16:33:54.886252 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:54.886217    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26"]
Sep 11 16:33:56.546019 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:56.545992    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:33:56.603840 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:33:56.603805    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" path="/var/lib/kubelet/pods/a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0/volumes"
Sep 11 16:34:26.547098 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:34:26.547053    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 11 16:34:36.547118 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:34:36.547076    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 11 16:34:46.547000 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:34:46.546965    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 11 16:34:56.546740 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:34:56.546698    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 11 16:35:06.594115 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:06.594077    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:35:12.737458 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:12.737419    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:13.956994 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:13.956962    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"]
Sep 11 16:35:13.957388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:13.957210    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kserve-container" containerID="cri-o://18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3" gracePeriod=30
Sep 11 16:35:13.957388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:13.957291    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kube-rbac-proxy" containerID="cri-o://f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3" gracePeriod=30
Sep 11 16:35:13.987752 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:13.987720    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:14.861438 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:14.861405    2576 generic.go:356] "Generic (PLEG): container finished" podID="e91f23cf-5020-4b33-957d-333e4115671a" containerID="f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3" exitCode=2
Sep 11 16:35:14.861438 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:14.861444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" event={"ID":"e91f23cf-5020-4b33-957d-333e4115671a","Type":"ContainerDied","Data":"f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3"}
Sep 11 16:35:16.542026 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:16.541984    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 11 16:35:16.547471 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:16.547443    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 11 16:35:18.596863 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.596841    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:35:18.637476 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.637452    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjmsf\" (UniqueName: \"kubernetes.io/projected/e91f23cf-5020-4b33-957d-333e4115671a-kube-api-access-cjmsf\") pod \"e91f23cf-5020-4b33-957d-333e4115671a\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") "
Sep 11 16:35:18.637628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.637491    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e91f23cf-5020-4b33-957d-333e4115671a-kserve-provision-location\") pod \"e91f23cf-5020-4b33-957d-333e4115671a\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") "
Sep 11 16:35:18.637628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.637531    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-kube-rbac-proxy-sar-config\") pod \"e91f23cf-5020-4b33-957d-333e4115671a\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") "
Sep 11 16:35:18.637628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.637575    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e91f23cf-5020-4b33-957d-333e4115671a-proxy-tls\") pod \"e91f23cf-5020-4b33-957d-333e4115671a\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") "
Sep 11 16:35:18.637628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.637593    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-cabundle-cert\") pod \"e91f23cf-5020-4b33-957d-333e4115671a\" (UID: \"e91f23cf-5020-4b33-957d-333e4115671a\") "
Sep 11 16:35:18.637874 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.637848    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91f23cf-5020-4b33-957d-333e4115671a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e91f23cf-5020-4b33-957d-333e4115671a" (UID: "e91f23cf-5020-4b33-957d-333e4115671a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:35:18.638033 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.638005    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e91f23cf-5020-4b33-957d-333e4115671a" (UID: "e91f23cf-5020-4b33-957d-333e4115671a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:35:18.638228 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.638048    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e91f23cf-5020-4b33-957d-333e4115671a" (UID: "e91f23cf-5020-4b33-957d-333e4115671a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:35:18.639800 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.639781    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91f23cf-5020-4b33-957d-333e4115671a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e91f23cf-5020-4b33-957d-333e4115671a" (UID: "e91f23cf-5020-4b33-957d-333e4115671a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:35:18.639910 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.639815    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91f23cf-5020-4b33-957d-333e4115671a-kube-api-access-cjmsf" (OuterVolumeSpecName: "kube-api-access-cjmsf") pod "e91f23cf-5020-4b33-957d-333e4115671a" (UID: "e91f23cf-5020-4b33-957d-333e4115671a"). InnerVolumeSpecName "kube-api-access-cjmsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:35:18.738765 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.738740    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:35:18.738765 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.738764    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e91f23cf-5020-4b33-957d-333e4115671a-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:35:18.738965 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.738774    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e91f23cf-5020-4b33-957d-333e4115671a-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:35:18.738965 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.738782    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjmsf\" (UniqueName: \"kubernetes.io/projected/e91f23cf-5020-4b33-957d-333e4115671a-kube-api-access-cjmsf\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:35:18.738965 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.738790    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e91f23cf-5020-4b33-957d-333e4115671a-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:35:18.878215 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.878181    2576 generic.go:356] "Generic (PLEG): container finished" podID="e91f23cf-5020-4b33-957d-333e4115671a" containerID="18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3" exitCode=0
Sep 11 16:35:18.878384 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.878237    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"
Sep 11 16:35:18.878384 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.878247    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" event={"ID":"e91f23cf-5020-4b33-957d-333e4115671a","Type":"ContainerDied","Data":"18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3"}
Sep 11 16:35:18.878384 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.878307    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl" event={"ID":"e91f23cf-5020-4b33-957d-333e4115671a","Type":"ContainerDied","Data":"a754188bb73aa673c118fe7561796c96638f0bb4237b15e92e874396529acbe0"}
Sep 11 16:35:18.878384 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.878343    2576 scope.go:117] "RemoveContainer" containerID="f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3"
Sep 11 16:35:18.886987 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.886968    2576 scope.go:117] "RemoveContainer" containerID="18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3"
Sep 11 16:35:18.894183 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.894164    2576 scope.go:117] "RemoveContainer" containerID="2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf"
Sep 11 16:35:18.901433 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.901418    2576 scope.go:117] "RemoveContainer" containerID="f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3"
Sep 11 16:35:18.901724 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:35:18.901706    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3\": container with ID starting with f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3 not found: ID does not exist" containerID="f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3"
Sep 11 16:35:18.901768 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.901733    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3"} err="failed to get container status \"f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3\": rpc error: code = NotFound desc = could not find container \"f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3\": container with ID starting with f6d4ca3ef86fc080142bf36f1187791c5728ac5e8da6f0ac33237a58a844ffb3 not found: ID does not exist"
Sep 11 16:35:18.901768 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.901749    2576 scope.go:117] "RemoveContainer" containerID="18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3"
Sep 11 16:35:18.902007 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:35:18.901989    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3\": container with ID starting with 18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3 not found: ID does not exist" containerID="18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3"
Sep 11 16:35:18.902064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.902012    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3"} err="failed to get container status \"18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3\": rpc error: code = NotFound desc = could not find container \"18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3\": container with ID starting with 18ef9f91154ec3121ffa6874f78860ec0e9ed64d20daf1955920500c1a9539d3 not found: ID does not exist"
Sep 11 16:35:18.902064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.902026    2576 scope.go:117] "RemoveContainer" containerID="2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf"
Sep 11 16:35:18.902265 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:35:18.902248    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf\": container with ID starting with 2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf not found: ID does not exist" containerID="2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf"
Sep 11 16:35:18.902304 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.902270    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf"} err="failed to get container status \"2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf\": rpc error: code = NotFound desc = could not find container \"2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf\": container with ID starting with 2dd2cfbc709cae16a3b549b5d25d2ded1c6706c34dbd4438ced8f58323628adf not found: ID does not exist"
Sep 11 16:35:18.913339 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.913321    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"]
Sep 11 16:35:18.917157 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:18.917137    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl"]
Sep 11 16:35:19.172336 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.172250    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:19.316006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.315958    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"]
Sep 11 16:35:19.316314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316298    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="storage-initializer"
Sep 11 16:35:19.316388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316317    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="storage-initializer"
Sep 11 16:35:19.316388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316329    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="storage-initializer"
Sep 11 16:35:19.316388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316338    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="storage-initializer"
Sep 11 16:35:19.316388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316353    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kube-rbac-proxy"
Sep 11 16:35:19.316388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316362    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kube-rbac-proxy"
Sep 11 16:35:19.316388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316376    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent"
Sep 11 16:35:19.316388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316384    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316398    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316407    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316419    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="storage-initializer"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316426    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="storage-initializer"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316439    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kserve-container"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316446    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kserve-container"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316462    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316470    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316483    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kube-rbac-proxy"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316491    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kube-rbac-proxy"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316502    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316510    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316580    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kserve-container"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316594    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="deeb805b-ce24-42c7-9f11-a9110dcc282a" containerName="kube-rbac-proxy"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316603    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kserve-container"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316612    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e91f23cf-5020-4b33-957d-333e4115671a" containerName="kube-rbac-proxy"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316619    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kserve-container"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316628    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="agent"
Sep 11 16:35:19.316846 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.316638    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a782a8f4-ba76-4486-b9f8-3c9f28d6a5c0" containerName="kube-rbac-proxy"
Sep 11 16:35:19.321453 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.321433    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.329983 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.329957    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 11 16:35:19.330214 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.330199    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:35:19.330315 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.330234    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:35:19.330315 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.330294    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:35:19.330532 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.330517    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 11 16:35:19.330598 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.330543    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:35:19.330598 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.330518    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:35:19.354920 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.354893    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"]
Sep 11 16:35:19.424670 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.424587    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:19.445826 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.445800    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.445980 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.445836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.445980 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.445869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.445980 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.445931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkb4f\" (UniqueName: \"kubernetes.io/projected/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-api-access-hkb4f\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.446383 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.446016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdeab8ca-2365-47b0-8e6c-332cf280c76e-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.546631 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.546592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.546813 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.546639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkb4f\" (UniqueName: \"kubernetes.io/projected/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-api-access-hkb4f\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.546813 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.546759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdeab8ca-2365-47b0-8e6c-332cf280c76e-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.546976 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.546918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.547046 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.546990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.547359 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.547335    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.547442 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.547348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.547541 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.547525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.549294 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.549273    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdeab8ca-2365-47b0-8e6c-332cf280c76e-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.560502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.560473    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkb4f\" (UniqueName: \"kubernetes.io/projected/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-api-access-hkb4f\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.631651 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.631611    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:19.915701 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:19.915666    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"]
Sep 11 16:35:19.918579 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:35:19.918552    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdeab8ca_2365_47b0_8e6c_332cf280c76e.slice/crio-6d41446c40d476c0a70fe02e566fff023c2cc1fad6e2924aaf8ba62295dcdd2a WatchSource:0}: Error finding container 6d41446c40d476c0a70fe02e566fff023c2cc1fad6e2924aaf8ba62295dcdd2a: Status 404 returned error can't find the container with id 6d41446c40d476c0a70fe02e566fff023c2cc1fad6e2924aaf8ba62295dcdd2a
Sep 11 16:35:20.602496 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:20.602463    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e91f23cf-5020-4b33-957d-333e4115671a" path="/var/lib/kubelet/pods/e91f23cf-5020-4b33-957d-333e4115671a/volumes"
Sep 11 16:35:20.891678 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:20.891591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" event={"ID":"bdeab8ca-2365-47b0-8e6c-332cf280c76e","Type":"ContainerStarted","Data":"7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9"}
Sep 11 16:35:20.891678 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:20.891659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" event={"ID":"bdeab8ca-2365-47b0-8e6c-332cf280c76e","Type":"ContainerStarted","Data":"6d41446c40d476c0a70fe02e566fff023c2cc1fad6e2924aaf8ba62295dcdd2a"}
Sep 11 16:35:23.905442 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:23.905359    2576 generic.go:356] "Generic (PLEG): container finished" podID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerID="7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9" exitCode=0
Sep 11 16:35:23.905442 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:23.905434    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" event={"ID":"bdeab8ca-2365-47b0-8e6c-332cf280c76e","Type":"ContainerDied","Data":"7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9"}
Sep 11 16:35:24.910864 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:24.910832    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" event={"ID":"bdeab8ca-2365-47b0-8e6c-332cf280c76e","Type":"ContainerStarted","Data":"1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb"}
Sep 11 16:35:24.911265 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:24.910889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" event={"ID":"bdeab8ca-2365-47b0-8e6c-332cf280c76e","Type":"ContainerStarted","Data":"cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99"}
Sep 11 16:35:24.911265 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:24.911174    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:24.938350 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:24.938297    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podStartSLOduration=2.309396193 podStartE2EDuration="5.938281135s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.628884942s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:35:24.935374779 +0000 UTC m=+1118.845175941" watchObservedRunningTime="2026-09-11 16:35:24.938281135 +0000 UTC m=+1118.848082296"
Sep 11 16:35:25.914602 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:25.914567    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:35:31.923814 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:35:31.923786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:36:01.925306 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:01.925261    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:36:11.924327 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:11.924246    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:36:21.924486 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:21.924446    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:36:29.157415 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:29.157376    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:30.886930 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:30.886890    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:31.924991 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:31.924930    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:36:33.598502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:33.598454    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:36:36.062024 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.061987    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:36.113374 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.113343    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"]
Sep 11 16:36:36.116769 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.116747    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.119451 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.119430    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 11 16:36:36.119551 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.119469    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 11 16:36:36.128624 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.128602    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"]
Sep 11 16:36:36.220648 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.220617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.220819 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.220657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvd6\" (UniqueName: \"kubernetes.io/projected/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-api-access-xmvd6\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.220819 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.220681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.220819 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.220706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.220819 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.220771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.321302 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.321206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvd6\" (UniqueName: \"kubernetes.io/projected/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-api-access-xmvd6\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.321302 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.321248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.321302 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.321271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.321640 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.321306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.321640 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.321379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.321640 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:36:36.321483    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 11 16:36:36.321640 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:36:36.321554    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-proxy-tls podName:d6b5f4b8-934b-4462-b4a6-1ab78b438f5f nodeName:}" failed. No retries permitted until 2026-09-11 16:36:36.821533329 +0000 UTC m=+1190.731334494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" (UID: "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 11 16:36:36.321845 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.321731    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.322081 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.322060    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.322123 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.322060    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.331238 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.331220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvd6\" (UniqueName: \"kubernetes.io/projected/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-api-access-xmvd6\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.557889 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.557856    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:36.825563 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.825531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:36.828071 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:36.828048    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ng6fr\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:37.026684 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:37.026648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:36:37.299585 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:37.299505    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"]
Sep 11 16:36:37.301968 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:36:37.301902    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b5f4b8_934b_4462_b4a6_1ab78b438f5f.slice/crio-74ced7453e581d1d9ea97dd02b375deccf19f8b1024f3f2fb037c09d1479f9b3 WatchSource:0}: Error finding container 74ced7453e581d1d9ea97dd02b375deccf19f8b1024f3f2fb037c09d1479f9b3: Status 404 returned error can't find the container with id 74ced7453e581d1d9ea97dd02b375deccf19f8b1024f3f2fb037c09d1479f9b3
Sep 11 16:36:38.190160 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:38.190123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" event={"ID":"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f","Type":"ContainerStarted","Data":"a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e"}
Sep 11 16:36:38.190324 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:38.190184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" event={"ID":"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f","Type":"ContainerStarted","Data":"74ced7453e581d1d9ea97dd02b375deccf19f8b1024f3f2fb037c09d1479f9b3"}
Sep 11 16:36:43.694503 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:43.694469    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:36:46.529305 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:46.529273    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:36:46.530912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:46.530891    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:36:46.535919 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:46.535902    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:36:46.537525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:46.537509    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:36:49.242017 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:49.241985    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"]
Sep 11 16:36:49.242436 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:49.242243    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container" containerID="cri-o://cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99" gracePeriod=30
Sep 11 16:36:49.242436 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:49.242315    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kube-rbac-proxy" containerID="cri-o://1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb" gracePeriod=30
Sep 11 16:36:49.252970 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:49.252922    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:50.236620 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:50.236589    2576 generic.go:356] "Generic (PLEG): container finished" podID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerID="1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb" exitCode=2
Sep 11 16:36:50.236784 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:50.236662    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" event={"ID":"bdeab8ca-2365-47b0-8e6c-332cf280c76e","Type":"ContainerDied","Data":"1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb"}
Sep 11 16:36:51.918664 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:51.918619    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 11 16:36:53.598966 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.598907    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 11 16:36:53.683116 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.683086    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:36:53.756138 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.756064    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kserve-provision-location\") pod \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") "
Sep 11 16:36:53.756138 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.756097    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkb4f\" (UniqueName: \"kubernetes.io/projected/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-api-access-hkb4f\") pod \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") "
Sep 11 16:36:53.756138 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.756128    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-rbac-proxy-sar-config\") pod \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") "
Sep 11 16:36:53.756405 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.756161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdeab8ca-2365-47b0-8e6c-332cf280c76e-proxy-tls\") pod \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") "
Sep 11 16:36:53.756405 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.756192    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-cabundle-cert\") pod \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\" (UID: \"bdeab8ca-2365-47b0-8e6c-332cf280c76e\") "
Sep 11 16:36:53.756517 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.756423    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdeab8ca-2365-47b0-8e6c-332cf280c76e" (UID: "bdeab8ca-2365-47b0-8e6c-332cf280c76e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:53.756517 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.756474    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bdeab8ca-2365-47b0-8e6c-332cf280c76e" (UID: "bdeab8ca-2365-47b0-8e6c-332cf280c76e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:36:53.756606 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.756572    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bdeab8ca-2365-47b0-8e6c-332cf280c76e" (UID: "bdeab8ca-2365-47b0-8e6c-332cf280c76e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:36:53.758337 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.758316    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-api-access-hkb4f" (OuterVolumeSpecName: "kube-api-access-hkb4f") pod "bdeab8ca-2365-47b0-8e6c-332cf280c76e" (UID: "bdeab8ca-2365-47b0-8e6c-332cf280c76e"). InnerVolumeSpecName "kube-api-access-hkb4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:36:53.758337 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.758329    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdeab8ca-2365-47b0-8e6c-332cf280c76e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bdeab8ca-2365-47b0-8e6c-332cf280c76e" (UID: "bdeab8ca-2365-47b0-8e6c-332cf280c76e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:36:53.856837 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.856796    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:36:53.856837 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.856833    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkb4f\" (UniqueName: \"kubernetes.io/projected/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-api-access-hkb4f\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:36:53.856837 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.856845    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:36:53.856837 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.856855    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdeab8ca-2365-47b0-8e6c-332cf280c76e-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:36:53.857130 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:53.856864    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdeab8ca-2365-47b0-8e6c-332cf280c76e-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:36:54.254154 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.254122    2576 generic.go:356] "Generic (PLEG): container finished" podID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerID="cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99" exitCode=0
Sep 11 16:36:54.254343 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.254162    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" event={"ID":"bdeab8ca-2365-47b0-8e6c-332cf280c76e","Type":"ContainerDied","Data":"cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99"}
Sep 11 16:36:54.254343 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.254191    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"
Sep 11 16:36:54.254343 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.254219    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz" event={"ID":"bdeab8ca-2365-47b0-8e6c-332cf280c76e","Type":"ContainerDied","Data":"6d41446c40d476c0a70fe02e566fff023c2cc1fad6e2924aaf8ba62295dcdd2a"}
Sep 11 16:36:54.254343 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.254247    2576 scope.go:117] "RemoveContainer" containerID="1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb"
Sep 11 16:36:54.262780 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.262761    2576 scope.go:117] "RemoveContainer" containerID="cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99"
Sep 11 16:36:54.270632 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.270614    2576 scope.go:117] "RemoveContainer" containerID="7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9"
Sep 11 16:36:54.278012 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.277998    2576 scope.go:117] "RemoveContainer" containerID="1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb"
Sep 11 16:36:54.278299 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:36:54.278278    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb\": container with ID starting with 1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb not found: ID does not exist" containerID="1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb"
Sep 11 16:36:54.278368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.278312    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb"} err="failed to get container status \"1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb\": rpc error: code = NotFound desc = could not find container \"1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb\": container with ID starting with 1d80c6290ce43bc6fd2ca5613bc83fc91fc3d8d494ebb0913160cabd5b3f53cb not found: ID does not exist"
Sep 11 16:36:54.278368 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.278336    2576 scope.go:117] "RemoveContainer" containerID="cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99"
Sep 11 16:36:54.278605 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:36:54.278588    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99\": container with ID starting with cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99 not found: ID does not exist" containerID="cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99"
Sep 11 16:36:54.278647 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.278611    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99"} err="failed to get container status \"cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99\": rpc error: code = NotFound desc = could not find container \"cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99\": container with ID starting with cbefe75de5ee3ae2b9bc5a0a42cbfed956d004747cc4e70d733608cc89acef99 not found: ID does not exist"
Sep 11 16:36:54.278647 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.278628    2576 scope.go:117] "RemoveContainer" containerID="7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9"
Sep 11 16:36:54.278918 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:36:54.278895    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9\": container with ID starting with 7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9 not found: ID does not exist" containerID="7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9"
Sep 11 16:36:54.279039 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.278923    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9"} err="failed to get container status \"7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9\": rpc error: code = NotFound desc = could not find container \"7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9\": container with ID starting with 7de8935ba4e6d008a565b46e6735b3331a731bb9fe99417c44a77b48bb6085a9 not found: ID does not exist"
Sep 11 16:36:54.282053 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.282031    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"]
Sep 11 16:36:54.288974 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.288913    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz"]
Sep 11 16:36:54.384058 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.384029    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:54.602960 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.602853    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" path="/var/lib/kubelet/pods/bdeab8ca-2365-47b0-8e6c-332cf280c76e/volumes"
Sep 11 16:36:54.899055 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:36:54.898949    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:07.303456 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:37:07.303422    2576 generic.go:356] "Generic (PLEG): container finished" podID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerID="a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e" exitCode=0
Sep 11 16:37:07.303456 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:37:07.303457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" event={"ID":"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f","Type":"ContainerDied","Data":"a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e"}
Sep 11 16:37:07.304673 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:37:07.304654    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:38:24.542713 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:38:24.542669    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:29.747906 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:38:29.747865    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:30.422612 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:38:30.422575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:31.734122 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:38:31.734087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:49.234137 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:38:49.234100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:04.199891 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.199845    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:04.238230 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.238196    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"]
Sep 11 16:39:04.238571 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.238519    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="storage-initializer"
Sep 11 16:39:04.238643 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.238576    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="storage-initializer"
Sep 11 16:39:04.238643 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.238617    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container"
Sep 11 16:39:04.238643 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.238624    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container"
Sep 11 16:39:04.238643 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.238636    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kube-rbac-proxy"
Sep 11 16:39:04.238643 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.238644    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kube-rbac-proxy"
Sep 11 16:39:04.238868 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.238729    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kserve-container"
Sep 11 16:39:04.238868 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.238743    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdeab8ca-2365-47b0-8e6c-332cf280c76e" containerName="kube-rbac-proxy"
Sep 11 16:39:04.295758 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.295715    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:04.407648 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.407599    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"]
Sep 11 16:39:04.408121 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.407701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.411677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.411637    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 11 16:39:04.411778 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.411741    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 11 16:39:04.508493 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.508401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2e1b8f-ea29-412c-9345-43776fd66c24-kserve-provision-location\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.508493 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.508466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.508690 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.508497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvw2\" (UniqueName: \"kubernetes.io/projected/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-api-access-kvvw2\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.508690 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.508541    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-cabundle-cert\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.508690 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.508592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2e1b8f-ea29-412c-9345-43776fd66c24-proxy-tls\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.609167 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.609131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-cabundle-cert\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.609356 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.609199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2e1b8f-ea29-412c-9345-43776fd66c24-proxy-tls\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.609356 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.609240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2e1b8f-ea29-412c-9345-43776fd66c24-kserve-provision-location\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.609356 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.609278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.609356 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.609304    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvw2\" (UniqueName: \"kubernetes.io/projected/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-api-access-kvvw2\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.609865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.609836    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2e1b8f-ea29-412c-9345-43776fd66c24-kserve-provision-location\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.610023 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.609999    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-cabundle-cert\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.610023 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.610013    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.612590 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.612568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2e1b8f-ea29-412c-9345-43776fd66c24-proxy-tls\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.620295 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.620257    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvw2\" (UniqueName: \"kubernetes.io/projected/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-api-access-kvvw2\") pod \"isvc-sklearn-predictor-9d87cbc64-2k995\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:04.719919 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:04.719878    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:18.924700 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:18.924675    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"]
Sep 11 16:39:19.003701 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:39:19.003671    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2e1b8f_ea29_412c_9345_43776fd66c24.slice/crio-9d2770600913de4333181ce30cacbb91a003be4e5a8a052cdf2893f5a7a8e3fa WatchSource:0}: Error finding container 9d2770600913de4333181ce30cacbb91a003be4e5a8a052cdf2893f5a7a8e3fa: Status 404 returned error can't find the container with id 9d2770600913de4333181ce30cacbb91a003be4e5a8a052cdf2893f5a7a8e3fa
Sep 11 16:39:19.858818 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:19.858781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" event={"ID":"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f","Type":"ContainerStarted","Data":"d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e"}
Sep 11 16:39:19.859153 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:19.858855    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" event={"ID":"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f","Type":"ContainerStarted","Data":"19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9"}
Sep 11 16:39:19.859153 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:19.858898    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:39:19.859153 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:19.858930    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:39:19.860639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:19.860617    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" event={"ID":"6f2e1b8f-ea29-412c-9345-43776fd66c24","Type":"ContainerStarted","Data":"1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a"}
Sep 11 16:39:19.860744 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:19.860659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" event={"ID":"6f2e1b8f-ea29-412c-9345-43776fd66c24","Type":"ContainerStarted","Data":"9d2770600913de4333181ce30cacbb91a003be4e5a8a052cdf2893f5a7a8e3fa"}
Sep 11 16:39:19.885066 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:19.885018    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" podStartSLOduration=2.770824642 podStartE2EDuration="2m43.88500486s" totalImagesPullingTime="2m12.076082675s" totalInitContainerRuntime="29.038097543s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:39:19.883551165 +0000 UTC m=+1353.793352337" watchObservedRunningTime="2026-09-11 16:39:19.88500486 +0000 UTC m=+1353.794806021"
Sep 11 16:39:23.877540 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:23.877507    2576 generic.go:356] "Generic (PLEG): container finished" podID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerID="1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a" exitCode=0
Sep 11 16:39:23.878010 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:23.877571    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" event={"ID":"6f2e1b8f-ea29-412c-9345-43776fd66c24","Type":"ContainerDied","Data":"1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a"}
Sep 11 16:39:24.884106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:24.884071    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" event={"ID":"6f2e1b8f-ea29-412c-9345-43776fd66c24","Type":"ContainerStarted","Data":"ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a"}
Sep 11 16:39:24.884477 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:24.884191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" event={"ID":"6f2e1b8f-ea29-412c-9345-43776fd66c24","Type":"ContainerStarted","Data":"e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11"}
Sep 11 16:39:24.884477 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:24.884377    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:24.921034 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:24.920980    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podStartSLOduration=16.778652313 podStartE2EDuration="20.920959949s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.142307636s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:39:24.915105737 +0000 UTC m=+1358.824906903" watchObservedRunningTime="2026-09-11 16:39:24.920959949 +0000 UTC m=+1358.830761103"
Sep 11 16:39:25.870094 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:25.870060    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:39:25.888550 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:25.888528    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:25.889528 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:25.889497    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:39:26.892886 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:26.892844    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:39:31.898233 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:31.898202    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:39:31.898894 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:31.898861    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:39:41.899774 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:41.899727    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:39:51.899103 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:39:51.899066    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:40:01.899053 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:01.899012    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:40:11.899771 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:11.899724    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:40:21.899159 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:21.899092    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:40:31.899291 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:31.899252    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 11 16:40:41.900134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:41.900058    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:40:44.364673 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:44.364639    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"]
Sep 11 16:40:44.365139 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:44.364891    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container" containerID="cri-o://e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11" gracePeriod=30
Sep 11 16:40:44.365139 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:44.364977    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kube-rbac-proxy" containerID="cri-o://ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a" gracePeriod=30
Sep 11 16:40:44.417140 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:44.417096    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:45.193389 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:45.193354    2576 generic.go:356] "Generic (PLEG): container finished" podID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerID="ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a" exitCode=2
Sep 11 16:40:45.193559 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:45.193420    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" event={"ID":"6f2e1b8f-ea29-412c-9345-43776fd66c24","Type":"ContainerDied","Data":"ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a"}
Sep 11 16:40:46.893404 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:46.893361    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 11 16:40:49.205690 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.205664    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:40:49.209203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.209177    2576 generic.go:356] "Generic (PLEG): container finished" podID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerID="e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11" exitCode=0
Sep 11 16:40:49.209321 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.209219    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"
Sep 11 16:40:49.209321 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.209235    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" event={"ID":"6f2e1b8f-ea29-412c-9345-43776fd66c24","Type":"ContainerDied","Data":"e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11"}
Sep 11 16:40:49.209321 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.209286    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995" event={"ID":"6f2e1b8f-ea29-412c-9345-43776fd66c24","Type":"ContainerDied","Data":"9d2770600913de4333181ce30cacbb91a003be4e5a8a052cdf2893f5a7a8e3fa"}
Sep 11 16:40:49.209321 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.209315    2576 scope.go:117] "RemoveContainer" containerID="ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a"
Sep 11 16:40:49.217207 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.217186    2576 scope.go:117] "RemoveContainer" containerID="e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11"
Sep 11 16:40:49.224282 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.224265    2576 scope.go:117] "RemoveContainer" containerID="1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a"
Sep 11 16:40:49.236561 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.236533    2576 scope.go:117] "RemoveContainer" containerID="ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a"
Sep 11 16:40:49.236852 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:40:49.236833    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a\": container with ID starting with ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a not found: ID does not exist" containerID="ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a"
Sep 11 16:40:49.236910 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.236861    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a"} err="failed to get container status \"ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a\": rpc error: code = NotFound desc = could not find container \"ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a\": container with ID starting with ad29a2a09dda2b97c7d5a66de4fe0612ce9d39f1c3d0997cdbffdbbd9e9be18a not found: ID does not exist"
Sep 11 16:40:49.236910 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.236882    2576 scope.go:117] "RemoveContainer" containerID="e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11"
Sep 11 16:40:49.237187 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:40:49.237171    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11\": container with ID starting with e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11 not found: ID does not exist" containerID="e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11"
Sep 11 16:40:49.237230 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.237193    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11"} err="failed to get container status \"e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11\": rpc error: code = NotFound desc = could not find container \"e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11\": container with ID starting with e988d4aa232d804d82e86b9ced191328f845c0dcebd65e411a9be89315d32e11 not found: ID does not exist"
Sep 11 16:40:49.237230 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.237205    2576 scope.go:117] "RemoveContainer" containerID="1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a"
Sep 11 16:40:49.237475 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:40:49.237457    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a\": container with ID starting with 1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a not found: ID does not exist" containerID="1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a"
Sep 11 16:40:49.237515 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.237480    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a"} err="failed to get container status \"1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a\": rpc error: code = NotFound desc = could not find container \"1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a\": container with ID starting with 1bfa86ff695de277bf39c9226b8ec476f2fde0752c142ba7e50d071bbf65985a not found: ID does not exist"
Sep 11 16:40:49.310246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.310152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2e1b8f-ea29-412c-9345-43776fd66c24-proxy-tls\") pod \"6f2e1b8f-ea29-412c-9345-43776fd66c24\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") "
Sep 11 16:40:49.310246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.310199    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvvw2\" (UniqueName: \"kubernetes.io/projected/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-api-access-kvvw2\") pod \"6f2e1b8f-ea29-412c-9345-43776fd66c24\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") "
Sep 11 16:40:49.310246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.310235    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2e1b8f-ea29-412c-9345-43776fd66c24-kserve-provision-location\") pod \"6f2e1b8f-ea29-412c-9345-43776fd66c24\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") "
Sep 11 16:40:49.310492 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.310258    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-rbac-proxy-sar-config\") pod \"6f2e1b8f-ea29-412c-9345-43776fd66c24\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") "
Sep 11 16:40:49.310492 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.310379    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-cabundle-cert\") pod \"6f2e1b8f-ea29-412c-9345-43776fd66c24\" (UID: \"6f2e1b8f-ea29-412c-9345-43776fd66c24\") "
Sep 11 16:40:49.310659 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.310619    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2e1b8f-ea29-412c-9345-43776fd66c24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f2e1b8f-ea29-412c-9345-43776fd66c24" (UID: "6f2e1b8f-ea29-412c-9345-43776fd66c24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:40:49.310659 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.310620    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f2e1b8f-ea29-412c-9345-43776fd66c24" (UID: "6f2e1b8f-ea29-412c-9345-43776fd66c24"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:40:49.310785 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.310718    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f2e1b8f-ea29-412c-9345-43776fd66c24" (UID: "6f2e1b8f-ea29-412c-9345-43776fd66c24"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:40:49.312545 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.312525    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-api-access-kvvw2" (OuterVolumeSpecName: "kube-api-access-kvvw2") pod "6f2e1b8f-ea29-412c-9345-43776fd66c24" (UID: "6f2e1b8f-ea29-412c-9345-43776fd66c24"). InnerVolumeSpecName "kube-api-access-kvvw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:40:49.312545 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.312534    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2e1b8f-ea29-412c-9345-43776fd66c24-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f2e1b8f-ea29-412c-9345-43776fd66c24" (UID: "6f2e1b8f-ea29-412c-9345-43776fd66c24"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:40:49.411069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.411035    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2e1b8f-ea29-412c-9345-43776fd66c24-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:40:49.411069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.411066    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:40:49.411288 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.411079    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2e1b8f-ea29-412c-9345-43776fd66c24-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:40:49.411288 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.411095    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2e1b8f-ea29-412c-9345-43776fd66c24-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:40:49.411288 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.411105    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvw2\" (UniqueName: \"kubernetes.io/projected/6f2e1b8f-ea29-412c-9345-43776fd66c24-kube-api-access-kvvw2\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:40:49.500357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.500315    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:49.537189 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.537151    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"]
Sep 11 16:40:49.543217 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.543169    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995"]
Sep 11 16:40:49.908717 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:49.908671    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:50.602315 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:50.602286    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" path="/var/lib/kubelet/pods/6f2e1b8f-ea29-412c-9345-43776fd66c24/volumes"
Sep 11 16:40:55.601775 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:40:55.601738    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:41:06.626710 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:06.626675    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"]
Sep 11 16:41:06.627244 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:06.626891    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="kserve-container" containerID="cri-o://19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9" gracePeriod=30
Sep 11 16:41:06.627244 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:06.626972    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="kube-rbac-proxy" containerID="cri-o://d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e" gracePeriod=30
Sep 11 16:41:06.645853 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:06.645817    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:07.282221 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:07.282187    2576 generic.go:356] "Generic (PLEG): container finished" podID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerID="d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e" exitCode=2
Sep 11 16:41:07.282221 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:07.282229    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" event={"ID":"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f","Type":"ContainerDied","Data":"d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e"}
Sep 11 16:41:07.966069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:07.966047    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:41:08.060864 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.060781    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-proxy-tls\") pod \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") "
Sep 11 16:41:08.060864 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.060830    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmvd6\" (UniqueName: \"kubernetes.io/projected/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-api-access-xmvd6\") pod \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") "
Sep 11 16:41:08.060864 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.060860    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-rbac-proxy-sar-config\") pod \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") "
Sep 11 16:41:08.061173 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.060916    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-cabundle-cert\") pod \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") "
Sep 11 16:41:08.061173 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.060962    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kserve-provision-location\") pod \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\" (UID: \"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f\") "
Sep 11 16:41:08.061273 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.061250    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" (UID: "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:41:08.061311 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.061274    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" (UID: "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:41:08.063187 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.063165    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" (UID: "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:41:08.063303 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.063246    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-api-access-xmvd6" (OuterVolumeSpecName: "kube-api-access-xmvd6") pod "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" (UID: "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f"). InnerVolumeSpecName "kube-api-access-xmvd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:41:08.065467 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.065439    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" (UID: "d6b5f4b8-934b-4462-b4a6-1ab78b438f5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:41:08.162171 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.162138    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmvd6\" (UniqueName: \"kubernetes.io/projected/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-api-access-xmvd6\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:41:08.162171 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.162164    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:41:08.162171 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.162175    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:41:08.162387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.162184    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:41:08.162387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.162194    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:41:08.288195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.288160    2576 generic.go:356] "Generic (PLEG): container finished" podID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerID="19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9" exitCode=0
Sep 11 16:41:08.288395 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.288228    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"
Sep 11 16:41:08.288395 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.288246    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" event={"ID":"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f","Type":"ContainerDied","Data":"19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9"}
Sep 11 16:41:08.288395 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.288309    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr" event={"ID":"d6b5f4b8-934b-4462-b4a6-1ab78b438f5f","Type":"ContainerDied","Data":"74ced7453e581d1d9ea97dd02b375deccf19f8b1024f3f2fb037c09d1479f9b3"}
Sep 11 16:41:08.288395 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.288335    2576 scope.go:117] "RemoveContainer" containerID="d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e"
Sep 11 16:41:08.296911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.296893    2576 scope.go:117] "RemoveContainer" containerID="19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9"
Sep 11 16:41:08.304340 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.304325    2576 scope.go:117] "RemoveContainer" containerID="a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e"
Sep 11 16:41:08.311375 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.311356    2576 scope.go:117] "RemoveContainer" containerID="d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e"
Sep 11 16:41:08.311709 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:41:08.311679    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e\": container with ID starting with d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e not found: ID does not exist" containerID="d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e"
Sep 11 16:41:08.311820 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.311718    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e"} err="failed to get container status \"d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e\": rpc error: code = NotFound desc = could not find container \"d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e\": container with ID starting with d824c8af1e0b8757333d522a4cf15a9d6e05e8df5c05fd0c0b7ed5a77361da2e not found: ID does not exist"
Sep 11 16:41:08.311820 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.311741    2576 scope.go:117] "RemoveContainer" containerID="19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9"
Sep 11 16:41:08.312186 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:41:08.312144    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9\": container with ID starting with 19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9 not found: ID does not exist" containerID="19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9"
Sep 11 16:41:08.312186 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.312166    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9"} err="failed to get container status \"19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9\": rpc error: code = NotFound desc = could not find container \"19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9\": container with ID starting with 19ccc12f18e3a334fa62e9240e40f7193665f6094db1dbd8a91a693218f6d1b9 not found: ID does not exist"
Sep 11 16:41:08.312186 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.312182    2576 scope.go:117] "RemoveContainer" containerID="a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e"
Sep 11 16:41:08.312426 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:41:08.312406    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e\": container with ID starting with a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e not found: ID does not exist" containerID="a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e"
Sep 11 16:41:08.312488 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.312432    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e"} err="failed to get container status \"a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e\": rpc error: code = NotFound desc = could not find container \"a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e\": container with ID starting with a0bacddff51578901ecb61b2659a163a9f83a8afc94a00a3ca4e075b54ca7b8e not found: ID does not exist"
Sep 11 16:41:08.313027 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.313006    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"]
Sep 11 16:41:08.320589 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.320536    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr"]
Sep 11 16:41:08.603487 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:08.603396    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" path="/var/lib/kubelet/pods/d6b5f4b8-934b-4462-b4a6-1ab78b438f5f/volumes"
Sep 11 16:41:11.791379 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.791343    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:11.835125 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835084    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"]
Sep 11 16:41:11.835442 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835425    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container"
Sep 11 16:41:11.835525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835445    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container"
Sep 11 16:41:11.835525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835459    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="kube-rbac-proxy"
Sep 11 16:41:11.835525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835467    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="kube-rbac-proxy"
Sep 11 16:41:11.835525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835476    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kube-rbac-proxy"
Sep 11 16:41:11.835525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835486    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kube-rbac-proxy"
Sep 11 16:41:11.835525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835506    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="storage-initializer"
Sep 11 16:41:11.835525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835514    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="storage-initializer"
Sep 11 16:41:11.835881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835531    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="storage-initializer"
Sep 11 16:41:11.835881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835539    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="storage-initializer"
Sep 11 16:41:11.835881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835549    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="kserve-container"
Sep 11 16:41:11.835881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835557    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="kserve-container"
Sep 11 16:41:11.835881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835628    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="kserve-container"
Sep 11 16:41:11.835881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835638    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b5f4b8-934b-4462-b4a6-1ab78b438f5f" containerName="kube-rbac-proxy"
Sep 11 16:41:11.835881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835648    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kube-rbac-proxy"
Sep 11 16:41:11.835881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.835658    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2e1b8f-ea29-412c-9345-43776fd66c24" containerName="kserve-container"
Sep 11 16:41:11.838993 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.838972    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:11.841889 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.841865    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:41:11.842017 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.841898    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:41:11.842102 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.842063    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:41:11.842376 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.842360    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:41:11.843015 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.842995    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 11 16:41:11.843099 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.843067    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:41:11.843099 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.843004    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 11 16:41:11.851706 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.851686    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"]
Sep 11 16:41:11.995208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.995170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cd55664-8127-40a1-a2aa-50de931bf041-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:11.995208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.995209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cd55664-8127-40a1-a2aa-50de931bf041-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:11.995478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.995237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:11.995478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.995268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:11.995478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:11.995340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcw9r\" (UniqueName: \"kubernetes.io/projected/1cd55664-8127-40a1-a2aa-50de931bf041-kube-api-access-hcw9r\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.096667 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.096552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcw9r\" (UniqueName: \"kubernetes.io/projected/1cd55664-8127-40a1-a2aa-50de931bf041-kube-api-access-hcw9r\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.096667 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.096627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cd55664-8127-40a1-a2aa-50de931bf041-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.096667 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.096647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cd55664-8127-40a1-a2aa-50de931bf041-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.096667 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.096675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.097080 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.096696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.097080 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:41:12.096802    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 11 16:41:12.097080 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:41:12.096875    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd55664-8127-40a1-a2aa-50de931bf041-proxy-tls podName:1cd55664-8127-40a1-a2aa-50de931bf041 nodeName:}" failed. No retries permitted until 2026-09-11 16:41:12.596854484 +0000 UTC m=+1466.506655643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1cd55664-8127-40a1-a2aa-50de931bf041-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" (UID: "1cd55664-8127-40a1-a2aa-50de931bf041") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 11 16:41:12.097080 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.097076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cd55664-8127-40a1-a2aa-50de931bf041-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.097503 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.097481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.097541 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.097481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.108036 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.108012    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcw9r\" (UniqueName: \"kubernetes.io/projected/1cd55664-8127-40a1-a2aa-50de931bf041-kube-api-access-hcw9r\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.233866 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.233831    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:12.599880 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.599829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cd55664-8127-40a1-a2aa-50de931bf041-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.602509 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.602481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cd55664-8127-40a1-a2aa-50de931bf041-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:12.690192 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.690155    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:12.749373 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:12.749338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:13.025136 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:13.025114    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"]
Sep 11 16:41:13.026798 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:41:13.026774    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd55664_8127_40a1_a2aa_50de931bf041.slice/crio-04ae6b93207f395c4dfdce1346facd5d913131bd0dc87468e58c3f94f6eb40e3 WatchSource:0}: Error finding container 04ae6b93207f395c4dfdce1346facd5d913131bd0dc87468e58c3f94f6eb40e3: Status 404 returned error can't find the container with id 04ae6b93207f395c4dfdce1346facd5d913131bd0dc87468e58c3f94f6eb40e3
Sep 11 16:41:13.308992 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:13.308883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" event={"ID":"1cd55664-8127-40a1-a2aa-50de931bf041","Type":"ContainerStarted","Data":"050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c"}
Sep 11 16:41:13.308992 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:13.308964    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" event={"ID":"1cd55664-8127-40a1-a2aa-50de931bf041","Type":"ContainerStarted","Data":"04ae6b93207f395c4dfdce1346facd5d913131bd0dc87468e58c3f94f6eb40e3"}
Sep 11 16:41:18.329285 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:18.329253    2576 generic.go:356] "Generic (PLEG): container finished" podID="1cd55664-8127-40a1-a2aa-50de931bf041" containerID="050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c" exitCode=0
Sep 11 16:41:18.329658 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:18.329319    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" event={"ID":"1cd55664-8127-40a1-a2aa-50de931bf041","Type":"ContainerDied","Data":"050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c"}
Sep 11 16:41:19.336097 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:19.336061    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" event={"ID":"1cd55664-8127-40a1-a2aa-50de931bf041","Type":"ContainerStarted","Data":"d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2"}
Sep 11 16:41:19.336473 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:19.336122    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" event={"ID":"1cd55664-8127-40a1-a2aa-50de931bf041","Type":"ContainerStarted","Data":"288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c"}
Sep 11 16:41:19.336473 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:19.336388    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:19.336473 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:19.336440    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:19.364133 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:19.364080    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" podStartSLOduration=3.526319321 podStartE2EDuration="8.364063271s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.83774395s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:41:19.362846437 +0000 UTC m=+1473.272647598" watchObservedRunningTime="2026-09-11 16:41:19.364063271 +0000 UTC m=+1473.273864434"
Sep 11 16:41:25.346582 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:25.346548    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:41:46.553424 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:46.553393    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:41:46.555565 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:46.555544    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:41:46.559872 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:46.559852    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:41:46.562069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:41:46.562053    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:42:54.606546 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:42:54.606513    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:43:05.645200 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:05.645142    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 16:43:08.484787 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:08.484742    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:08.896026 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:08.895919    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"]
Sep 11 16:43:08.896205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:08.896180    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kserve-container" containerID="cri-o://288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c" gracePeriod=30
Sep 11 16:43:08.896279 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:08.896218    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kube-rbac-proxy" containerID="cri-o://d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2" gracePeriod=30
Sep 11 16:43:08.912113 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:08.912090    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:09.748115 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:09.748079    2576 generic.go:356] "Generic (PLEG): container finished" podID="1cd55664-8127-40a1-a2aa-50de931bf041" containerID="d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2" exitCode=2
Sep 11 16:43:09.748501 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:09.748151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" event={"ID":"1cd55664-8127-40a1-a2aa-50de931bf041","Type":"ContainerDied","Data":"d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2"}
Sep 11 16:43:10.341791 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.341478    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 11 16:43:10.443277 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.443253    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:43:10.475468 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.475435    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cd55664-8127-40a1-a2aa-50de931bf041-proxy-tls\") pod \"1cd55664-8127-40a1-a2aa-50de931bf041\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") "
Sep 11 16:43:10.475663 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.475477    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-cabundle-cert\") pod \"1cd55664-8127-40a1-a2aa-50de931bf041\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") "
Sep 11 16:43:10.475663 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.475601    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcw9r\" (UniqueName: \"kubernetes.io/projected/1cd55664-8127-40a1-a2aa-50de931bf041-kube-api-access-hcw9r\") pod \"1cd55664-8127-40a1-a2aa-50de931bf041\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") "
Sep 11 16:43:10.475663 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.475653    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cd55664-8127-40a1-a2aa-50de931bf041-kserve-provision-location\") pod \"1cd55664-8127-40a1-a2aa-50de931bf041\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") "
Sep 11 16:43:10.475848 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.475678    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-kube-rbac-proxy-sar-config\") pod \"1cd55664-8127-40a1-a2aa-50de931bf041\" (UID: \"1cd55664-8127-40a1-a2aa-50de931bf041\") "
Sep 11 16:43:10.475848 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.475832    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1cd55664-8127-40a1-a2aa-50de931bf041" (UID: "1cd55664-8127-40a1-a2aa-50de931bf041"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:43:10.476081 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.476059    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:43:10.476224 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.476121    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1cd55664-8127-40a1-a2aa-50de931bf041" (UID: "1cd55664-8127-40a1-a2aa-50de931bf041"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:43:10.477715 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.477692    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd55664-8127-40a1-a2aa-50de931bf041-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1cd55664-8127-40a1-a2aa-50de931bf041" (UID: "1cd55664-8127-40a1-a2aa-50de931bf041"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:43:10.478141 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.478115    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd55664-8127-40a1-a2aa-50de931bf041-kube-api-access-hcw9r" (OuterVolumeSpecName: "kube-api-access-hcw9r") pod "1cd55664-8127-40a1-a2aa-50de931bf041" (UID: "1cd55664-8127-40a1-a2aa-50de931bf041"). InnerVolumeSpecName "kube-api-access-hcw9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:43:10.481413 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.481354    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd55664-8127-40a1-a2aa-50de931bf041-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1cd55664-8127-40a1-a2aa-50de931bf041" (UID: "1cd55664-8127-40a1-a2aa-50de931bf041"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:10.577238 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.577201    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cd55664-8127-40a1-a2aa-50de931bf041-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:43:10.577238 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.577233    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcw9r\" (UniqueName: \"kubernetes.io/projected/1cd55664-8127-40a1-a2aa-50de931bf041-kube-api-access-hcw9r\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:43:10.577238 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.577245    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1cd55664-8127-40a1-a2aa-50de931bf041-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:43:10.577477 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.577254    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cd55664-8127-40a1-a2aa-50de931bf041-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:43:10.753844 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.753751    2576 generic.go:356] "Generic (PLEG): container finished" podID="1cd55664-8127-40a1-a2aa-50de931bf041" containerID="288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c" exitCode=0
Sep 11 16:43:10.753844 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.753810    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" event={"ID":"1cd55664-8127-40a1-a2aa-50de931bf041","Type":"ContainerDied","Data":"288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c"}
Sep 11 16:43:10.753844 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.753817    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"
Sep 11 16:43:10.754427 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.753860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2" event={"ID":"1cd55664-8127-40a1-a2aa-50de931bf041","Type":"ContainerDied","Data":"04ae6b93207f395c4dfdce1346facd5d913131bd0dc87468e58c3f94f6eb40e3"}
Sep 11 16:43:10.754427 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.753893    2576 scope.go:117] "RemoveContainer" containerID="d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2"
Sep 11 16:43:10.761739 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.761721    2576 scope.go:117] "RemoveContainer" containerID="288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c"
Sep 11 16:43:10.769387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.769361    2576 scope.go:117] "RemoveContainer" containerID="050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c"
Sep 11 16:43:10.776580 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.776555    2576 scope.go:117] "RemoveContainer" containerID="d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2"
Sep 11 16:43:10.776858 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:43:10.776835    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2\": container with ID starting with d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2 not found: ID does not exist" containerID="d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2"
Sep 11 16:43:10.776959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.776872    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2"} err="failed to get container status \"d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2\": rpc error: code = NotFound desc = could not find container \"d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2\": container with ID starting with d911f951988f6afc8d2e52f10688d2279537a575dbe1ce6eb4dbe1b8aacd70c2 not found: ID does not exist"
Sep 11 16:43:10.776959 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.776897    2576 scope.go:117] "RemoveContainer" containerID="288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c"
Sep 11 16:43:10.777266 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:43:10.777238    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c\": container with ID starting with 288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c not found: ID does not exist" containerID="288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c"
Sep 11 16:43:10.777354 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.777274    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c"} err="failed to get container status \"288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c\": rpc error: code = NotFound desc = could not find container \"288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c\": container with ID starting with 288e8d168def070b3df2966795be8958aa7e0c9ccee1478dc277f809a729c00c not found: ID does not exist"
Sep 11 16:43:10.777354 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.777295    2576 scope.go:117] "RemoveContainer" containerID="050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c"
Sep 11 16:43:10.777663 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:43:10.777642    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c\": container with ID starting with 050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c not found: ID does not exist" containerID="050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c"
Sep 11 16:43:10.777738 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.777669    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c"} err="failed to get container status \"050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c\": rpc error: code = NotFound desc = could not find container \"050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c\": container with ID starting with 050e0a6c5cd55267183062aa19859043995ed872b9bc260af97445e89e63089c not found: ID does not exist"
Sep 11 16:43:10.780081 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.780060    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"]
Sep 11 16:43:10.782230 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:10.782209    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2"]
Sep 11 16:43:12.603424 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:12.603393    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" path="/var/lib/kubelet/pods/1cd55664-8127-40a1-a2aa-50de931bf041/volumes"
Sep 11 16:43:14.077159 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.077110    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:14.230287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.230250    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"]
Sep 11 16:43:14.230595 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.230582    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kserve-container"
Sep 11 16:43:14.230639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.230597    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kserve-container"
Sep 11 16:43:14.230639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.230604    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kube-rbac-proxy"
Sep 11 16:43:14.230639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.230610    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kube-rbac-proxy"
Sep 11 16:43:14.230639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.230626    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="storage-initializer"
Sep 11 16:43:14.230639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.230635    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="storage-initializer"
Sep 11 16:43:14.230793 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.230681    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kube-rbac-proxy"
Sep 11 16:43:14.230793 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.230691    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd55664-8127-40a1-a2aa-50de931bf041" containerName="kserve-container"
Sep 11 16:43:14.235580 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.235561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.247381 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.247360    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:43:14.247514 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.247434    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:43:14.248550 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.248532    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:43:14.248550 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.248542    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 11 16:43:14.248713 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.248552    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:43:14.248713 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.248573    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:43:14.248805 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.248724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 11 16:43:14.255620 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.255600    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"]
Sep 11 16:43:14.305257 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.305221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.305257 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.305257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.305503 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.305335    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.305503 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.305380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4871cb8-cee5-45e4-bfc9-b53b6680233a-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.305503 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.305428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gmgf\" (UniqueName: \"kubernetes.io/projected/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-api-access-5gmgf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.406531 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.406440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4871cb8-cee5-45e4-bfc9-b53b6680233a-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.406531 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.406508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gmgf\" (UniqueName: \"kubernetes.io/projected/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-api-access-5gmgf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.406775 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.406545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.406775 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.406573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.406775 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:43:14.406596    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 11 16:43:14.406775 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.406631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.406775 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:43:14.406671    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4871cb8-cee5-45e4-bfc9-b53b6680233a-proxy-tls podName:e4871cb8-cee5-45e4-bfc9-b53b6680233a nodeName:}" failed. No retries permitted until 2026-09-11 16:43:14.906648421 +0000 UTC m=+1588.816449561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4871cb8-cee5-45e4-bfc9-b53b6680233a-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" (UID: "e4871cb8-cee5-45e4-bfc9-b53b6680233a") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 11 16:43:14.407136 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.407115    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.407357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.407342    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.407394 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.407360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.422991 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.422958    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gmgf\" (UniqueName: \"kubernetes.io/projected/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-api-access-5gmgf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.537160 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.537121    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:14.909509 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.909467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4871cb8-cee5-45e4-bfc9-b53b6680233a-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:14.912032 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:14.912010    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4871cb8-cee5-45e4-bfc9-b53b6680233a-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:15.145019 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:15.144977    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:15.427381 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:15.427356    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"]
Sep 11 16:43:15.429025 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:43:15.429000    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4871cb8_cee5_45e4_bfc9_b53b6680233a.slice/crio-e0abc8a1f2d3062a239584bfce18eff71ae44ea73cba4fbc70e8597500f57791 WatchSource:0}: Error finding container e0abc8a1f2d3062a239584bfce18eff71ae44ea73cba4fbc70e8597500f57791: Status 404 returned error can't find the container with id e0abc8a1f2d3062a239584bfce18eff71ae44ea73cba4fbc70e8597500f57791
Sep 11 16:43:15.776521 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:15.776430    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" event={"ID":"e4871cb8-cee5-45e4-bfc9-b53b6680233a","Type":"ContainerStarted","Data":"21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39"}
Sep 11 16:43:15.776521 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:15.776498    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" event={"ID":"e4871cb8-cee5-45e4-bfc9-b53b6680233a","Type":"ContainerStarted","Data":"e0abc8a1f2d3062a239584bfce18eff71ae44ea73cba4fbc70e8597500f57791"}
Sep 11 16:43:19.894620 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:19.894589    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:44.885705 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:44.885668    2576 generic.go:356] "Generic (PLEG): container finished" podID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerID="21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39" exitCode=0
Sep 11 16:43:44.886147 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:44.885724    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" event={"ID":"e4871cb8-cee5-45e4-bfc9-b53b6680233a","Type":"ContainerDied","Data":"21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39"}
Sep 11 16:43:45.891678 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:45.891645    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" event={"ID":"e4871cb8-cee5-45e4-bfc9-b53b6680233a","Type":"ContainerStarted","Data":"f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9"}
Sep 11 16:43:45.892097 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:45.891706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" event={"ID":"e4871cb8-cee5-45e4-bfc9-b53b6680233a","Type":"ContainerStarted","Data":"975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c"}
Sep 11 16:43:45.892097 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:45.891897    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:45.916610 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:45.916559    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" podStartSLOduration=3.401456112 podStartE2EDuration="31.916542306s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.515086194s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:45.915078929 +0000 UTC m=+1619.824880091" watchObservedRunningTime="2026-09-11 16:43:45.916542306 +0000 UTC m=+1619.826343467"
Sep 11 16:43:46.903240 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:46.903135    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:52.912374 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:52.912346    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:43:59.834282 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:43:59.834246    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:04.938876 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:04.938792    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:04.985176 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:04.985145    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"]
Sep 11 16:44:04.991378 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:04.991357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:04.994226 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:04.994205    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 11 16:44:04.994477 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:04.994463    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 11 16:44:05.000401 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.000379    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"]
Sep 11 16:44:05.049500 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.049470    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:05.102120 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.102088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zn88\" (UniqueName: \"kubernetes.io/projected/4f348118-9023-47bb-baae-59061bc7b9da-kube-api-access-4zn88\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.102317 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.102159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f348118-9023-47bb-baae-59061bc7b9da-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.102317 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.102200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f348118-9023-47bb-baae-59061bc7b9da-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.102317 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.102244    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.102317 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.102280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.203569 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.203543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.203728 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.203580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zn88\" (UniqueName: \"kubernetes.io/projected/4f348118-9023-47bb-baae-59061bc7b9da-kube-api-access-4zn88\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.203728 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.203618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f348118-9023-47bb-baae-59061bc7b9da-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.203728 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.203636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f348118-9023-47bb-baae-59061bc7b9da-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.203728 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.203659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.204228 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.204192    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f348118-9023-47bb-baae-59061bc7b9da-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.204377 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.204353    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.204426 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.204408    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.206499 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.206473    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f348118-9023-47bb-baae-59061bc7b9da-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.215407 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.215383    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zn88\" (UniqueName: \"kubernetes.io/projected/4f348118-9023-47bb-baae-59061bc7b9da-kube-api-access-4zn88\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.302338 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.302296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:05.574970 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.574927    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"]
Sep 11 16:44:05.576883 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:44:05.576857    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f348118_9023_47bb_baae_59061bc7b9da.slice/crio-70bc0d564f5160f59c59e3131d741bc1c984ce5ea42df4e5a0f062db554400b9 WatchSource:0}: Error finding container 70bc0d564f5160f59c59e3131d741bc1c984ce5ea42df4e5a0f062db554400b9: Status 404 returned error can't find the container with id 70bc0d564f5160f59c59e3131d741bc1c984ce5ea42df4e5a0f062db554400b9
Sep 11 16:44:05.974892 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.974857    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" event={"ID":"4f348118-9023-47bb-baae-59061bc7b9da","Type":"ContainerStarted","Data":"ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f"}
Sep 11 16:44:05.975263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:05.974919    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" event={"ID":"4f348118-9023-47bb-baae-59061bc7b9da","Type":"ContainerStarted","Data":"70bc0d564f5160f59c59e3131d741bc1c984ce5ea42df4e5a0f062db554400b9"}
Sep 11 16:44:13.001520 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:13.001482    2576 generic.go:356] "Generic (PLEG): container finished" podID="4f348118-9023-47bb-baae-59061bc7b9da" containerID="ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f" exitCode=0
Sep 11 16:44:13.001912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:13.001554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" event={"ID":"4f348118-9023-47bb-baae-59061bc7b9da","Type":"ContainerDied","Data":"ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f"}
Sep 11 16:44:14.012288 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:14.012241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" event={"ID":"4f348118-9023-47bb-baae-59061bc7b9da","Type":"ContainerStarted","Data":"1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d"}
Sep 11 16:44:14.012780 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:14.012334    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" event={"ID":"4f348118-9023-47bb-baae-59061bc7b9da","Type":"ContainerStarted","Data":"093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006"}
Sep 11 16:44:14.012780 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:14.012624    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:14.039319 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:14.039265    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" podStartSLOduration=3.207964645 podStartE2EDuration="10.039250994s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.831286349s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:14.038200633 +0000 UTC m=+1647.948001796" watchObservedRunningTime="2026-09-11 16:44:14.039250994 +0000 UTC m=+1647.949052193"
Sep 11 16:44:15.016860 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:15.016820    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:15.018385 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:15.018354    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:44:16.020602 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:16.020558    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:44:21.025224 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:21.025192    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:21.025848 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:21.025814    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 11 16:44:31.026052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:31.026021    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:36.921317 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:36.921277    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"]
Sep 11 16:44:36.921742 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:36.921545    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kserve-container" containerID="cri-o://093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006" gracePeriod=30
Sep 11 16:44:36.921742 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:36.921605    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kube-rbac-proxy" containerID="cri-o://1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d" gracePeriod=30
Sep 11 16:44:36.930813 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:36.930785    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:37.115358 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:37.115329    2576 generic.go:356] "Generic (PLEG): container finished" podID="4f348118-9023-47bb-baae-59061bc7b9da" containerID="1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d" exitCode=2
Sep 11 16:44:37.115510 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:37.115369    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" event={"ID":"4f348118-9023-47bb-baae-59061bc7b9da","Type":"ContainerDied","Data":"1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d"}
Sep 11 16:44:37.967210 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:37.967187    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:38.056242 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.056150    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f348118-9023-47bb-baae-59061bc7b9da-proxy-tls\") pod \"4f348118-9023-47bb-baae-59061bc7b9da\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") "
Sep 11 16:44:38.056242 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.056189    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zn88\" (UniqueName: \"kubernetes.io/projected/4f348118-9023-47bb-baae-59061bc7b9da-kube-api-access-4zn88\") pod \"4f348118-9023-47bb-baae-59061bc7b9da\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") "
Sep 11 16:44:38.056242 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.056212    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f348118-9023-47bb-baae-59061bc7b9da-kserve-provision-location\") pod \"4f348118-9023-47bb-baae-59061bc7b9da\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") "
Sep 11 16:44:38.056242 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.056230    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-cabundle-cert\") pod \"4f348118-9023-47bb-baae-59061bc7b9da\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") "
Sep 11 16:44:38.056242 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.056246    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-kube-rbac-proxy-sar-config\") pod \"4f348118-9023-47bb-baae-59061bc7b9da\" (UID: \"4f348118-9023-47bb-baae-59061bc7b9da\") "
Sep 11 16:44:38.056636 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.056604    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f348118-9023-47bb-baae-59061bc7b9da" (UID: "4f348118-9023-47bb-baae-59061bc7b9da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:44:38.056686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.056642    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f348118-9023-47bb-baae-59061bc7b9da" (UID: "4f348118-9023-47bb-baae-59061bc7b9da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:44:38.058506 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.058475    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f348118-9023-47bb-baae-59061bc7b9da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f348118-9023-47bb-baae-59061bc7b9da" (UID: "4f348118-9023-47bb-baae-59061bc7b9da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:44:38.058643 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.058622    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f348118-9023-47bb-baae-59061bc7b9da-kube-api-access-4zn88" (OuterVolumeSpecName: "kube-api-access-4zn88") pod "4f348118-9023-47bb-baae-59061bc7b9da" (UID: "4f348118-9023-47bb-baae-59061bc7b9da"). InnerVolumeSpecName "kube-api-access-4zn88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:44:38.078526 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.078503    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f348118-9023-47bb-baae-59061bc7b9da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f348118-9023-47bb-baae-59061bc7b9da" (UID: "4f348118-9023-47bb-baae-59061bc7b9da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:44:38.121530 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.121499    2576 generic.go:356] "Generic (PLEG): container finished" podID="4f348118-9023-47bb-baae-59061bc7b9da" containerID="093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006" exitCode=0
Sep 11 16:44:38.121658 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.121535    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" event={"ID":"4f348118-9023-47bb-baae-59061bc7b9da","Type":"ContainerDied","Data":"093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006"}
Sep 11 16:44:38.121658 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.121557    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"
Sep 11 16:44:38.121658 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.121582    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2" event={"ID":"4f348118-9023-47bb-baae-59061bc7b9da","Type":"ContainerDied","Data":"70bc0d564f5160f59c59e3131d741bc1c984ce5ea42df4e5a0f062db554400b9"}
Sep 11 16:44:38.121658 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.121608    2576 scope.go:117] "RemoveContainer" containerID="1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d"
Sep 11 16:44:38.130189 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.130173    2576 scope.go:117] "RemoveContainer" containerID="093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006"
Sep 11 16:44:38.137791 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.137775    2576 scope.go:117] "RemoveContainer" containerID="ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f"
Sep 11 16:44:38.145114 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.145045    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"]
Sep 11 16:44:38.146032 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.146013    2576 scope.go:117] "RemoveContainer" containerID="1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d"
Sep 11 16:44:38.146352 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:44:38.146330    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d\": container with ID starting with 1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d not found: ID does not exist" containerID="1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d"
Sep 11 16:44:38.146406 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.146362    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d"} err="failed to get container status \"1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d\": rpc error: code = NotFound desc = could not find container \"1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d\": container with ID starting with 1fe9a0ab476df41ac2fe1ec00b1b609d0816e83406ca69b7ad7361f23fa7709d not found: ID does not exist"
Sep 11 16:44:38.146406 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.146379    2576 scope.go:117] "RemoveContainer" containerID="093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006"
Sep 11 16:44:38.146657 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.146621    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2"]
Sep 11 16:44:38.146657 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:44:38.146618    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006\": container with ID starting with 093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006 not found: ID does not exist" containerID="093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006"
Sep 11 16:44:38.146758 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.146672    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006"} err="failed to get container status \"093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006\": rpc error: code = NotFound desc = could not find container \"093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006\": container with ID starting with 093bed6125b130de349de09573e6ac3bddd07d96320971384e499880308a3006 not found: ID does not exist"
Sep 11 16:44:38.146758 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.146689    2576 scope.go:117] "RemoveContainer" containerID="ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f"
Sep 11 16:44:38.146961 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:44:38.146919    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f\": container with ID starting with ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f not found: ID does not exist" containerID="ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f"
Sep 11 16:44:38.147013 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.146963    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f"} err="failed to get container status \"ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f\": rpc error: code = NotFound desc = could not find container \"ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f\": container with ID starting with ddaa02136e9bae4ca6838cab67504ed6829ce89d5fbe9415115d53e19d3bd17f not found: ID does not exist"
Sep 11 16:44:38.157691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.157671    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f348118-9023-47bb-baae-59061bc7b9da-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:44:38.157691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.157689    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zn88\" (UniqueName: \"kubernetes.io/projected/4f348118-9023-47bb-baae-59061bc7b9da-kube-api-access-4zn88\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:44:38.157797 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.157698    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f348118-9023-47bb-baae-59061bc7b9da-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:44:38.157797 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.157706    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:44:38.157797 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.157714    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f348118-9023-47bb-baae-59061bc7b9da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:44:38.602188 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:38.602147    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f348118-9023-47bb-baae-59061bc7b9da" path="/var/lib/kubelet/pods/4f348118-9023-47bb-baae-59061bc7b9da/volumes"
Sep 11 16:44:42.048502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:42.048459    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:42.188246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:44:42.188208    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:23.601837 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:23.601801    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:45:30.398537 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:30.398504    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"]
Sep 11 16:45:30.398974 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:30.398752    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="kserve-container" containerID="cri-o://975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c" gracePeriod=30
Sep 11 16:45:30.398974 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:30.398785    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="kube-rbac-proxy" containerID="cri-o://f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9" gracePeriod=30
Sep 11 16:45:30.407850 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:30.407824    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:31.327506 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.327473    2576 generic.go:356] "Generic (PLEG): container finished" podID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerID="f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9" exitCode=2
Sep 11 16:45:31.327707 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.327549    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" event={"ID":"e4871cb8-cee5-45e4-bfc9-b53b6680233a","Type":"ContainerDied","Data":"f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9"}
Sep 11 16:45:31.742588 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.742567    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:45:31.777221 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.777197    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-cabundle-cert\") pod \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") "
Sep 11 16:45:31.777370 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.777233    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gmgf\" (UniqueName: \"kubernetes.io/projected/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-api-access-5gmgf\") pod \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") "
Sep 11 16:45:31.777370 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.777272    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-rbac-proxy-sar-config\") pod \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") "
Sep 11 16:45:31.777370 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.777307    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4871cb8-cee5-45e4-bfc9-b53b6680233a-proxy-tls\") pod \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") "
Sep 11 16:45:31.777370 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.777355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kserve-provision-location\") pod \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\" (UID: \"e4871cb8-cee5-45e4-bfc9-b53b6680233a\") "
Sep 11 16:45:31.777594 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.777565    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4871cb8-cee5-45e4-bfc9-b53b6680233a" (UID: "e4871cb8-cee5-45e4-bfc9-b53b6680233a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:45:31.777810 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.777776    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4871cb8-cee5-45e4-bfc9-b53b6680233a" (UID: "e4871cb8-cee5-45e4-bfc9-b53b6680233a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:45:31.779539 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.779510    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4871cb8-cee5-45e4-bfc9-b53b6680233a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4871cb8-cee5-45e4-bfc9-b53b6680233a" (UID: "e4871cb8-cee5-45e4-bfc9-b53b6680233a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:45:31.779539 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.779518    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-api-access-5gmgf" (OuterVolumeSpecName: "kube-api-access-5gmgf") pod "e4871cb8-cee5-45e4-bfc9-b53b6680233a" (UID: "e4871cb8-cee5-45e4-bfc9-b53b6680233a"). InnerVolumeSpecName "kube-api-access-5gmgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:45:31.782136 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.782113    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4871cb8-cee5-45e4-bfc9-b53b6680233a" (UID: "e4871cb8-cee5-45e4-bfc9-b53b6680233a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:45:31.878389 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.878313    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:45:31.878389 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.878338    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gmgf\" (UniqueName: \"kubernetes.io/projected/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-api-access-5gmgf\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:45:31.878389 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.878348    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:45:31.878389 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.878357    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4871cb8-cee5-45e4-bfc9-b53b6680233a-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:45:31.878389 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:31.878365    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4871cb8-cee5-45e4-bfc9-b53b6680233a-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:45:32.333680 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.333644    2576 generic.go:356] "Generic (PLEG): container finished" podID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerID="975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c" exitCode=0
Sep 11 16:45:32.333875 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.333700    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"
Sep 11 16:45:32.333875 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.333727    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" event={"ID":"e4871cb8-cee5-45e4-bfc9-b53b6680233a","Type":"ContainerDied","Data":"975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c"}
Sep 11 16:45:32.333875 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.333799    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k" event={"ID":"e4871cb8-cee5-45e4-bfc9-b53b6680233a","Type":"ContainerDied","Data":"e0abc8a1f2d3062a239584bfce18eff71ae44ea73cba4fbc70e8597500f57791"}
Sep 11 16:45:32.333875 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.333836    2576 scope.go:117] "RemoveContainer" containerID="f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9"
Sep 11 16:45:32.341880 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.341861    2576 scope.go:117] "RemoveContainer" containerID="975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c"
Sep 11 16:45:32.349261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.349244    2576 scope.go:117] "RemoveContainer" containerID="21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39"
Sep 11 16:45:32.356511 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.356420    2576 scope.go:117] "RemoveContainer" containerID="f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9"
Sep 11 16:45:32.358863 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:45:32.358834    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9\": container with ID starting with f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9 not found: ID does not exist" containerID="f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9"
Sep 11 16:45:32.358997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.358872    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9"} err="failed to get container status \"f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9\": rpc error: code = NotFound desc = could not find container \"f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9\": container with ID starting with f67f5ca61a55f6c2b6d24d895ef93a2e4529ab586f646f1b61c037b4e596c6a9 not found: ID does not exist"
Sep 11 16:45:32.358997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.358895    2576 scope.go:117] "RemoveContainer" containerID="975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c"
Sep 11 16:45:32.359518 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:45:32.359457    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c\": container with ID starting with 975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c not found: ID does not exist" containerID="975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c"
Sep 11 16:45:32.359604 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.359553    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c"} err="failed to get container status \"975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c\": rpc error: code = NotFound desc = could not find container \"975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c\": container with ID starting with 975c8ece665fd3239888b711d2d216a2c1fbd1bcb8bccffea303eb42989cbb2c not found: ID does not exist"
Sep 11 16:45:32.359604 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.359577    2576 scope.go:117] "RemoveContainer" containerID="21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39"
Sep 11 16:45:32.359901 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:45:32.359881    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39\": container with ID starting with 21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39 not found: ID does not exist" containerID="21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39"
Sep 11 16:45:32.360010 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.359910    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39"} err="failed to get container status \"21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39\": rpc error: code = NotFound desc = could not find container \"21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39\": container with ID starting with 21837f6d3784e714cec5c6d03ef6098893d0c8a9a85f5a5bac09240aa03afa39 not found: ID does not exist"
Sep 11 16:45:32.361277 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.361260    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"]
Sep 11 16:45:32.366807 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.366787    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k"]
Sep 11 16:45:32.602746 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:32.602663    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" path="/var/lib/kubelet/pods/e4871cb8-cee5-45e4-bfc9-b53b6680233a/volumes"
Sep 11 16:45:35.541510 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.541473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:35.609647 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609614    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"]
Sep 11 16:45:35.609903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609890    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kserve-container"
Sep 11 16:45:35.609972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609905    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kserve-container"
Sep 11 16:45:35.609972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609914    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="kube-rbac-proxy"
Sep 11 16:45:35.609972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609919    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="kube-rbac-proxy"
Sep 11 16:45:35.609972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609929    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="storage-initializer"
Sep 11 16:45:35.609972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609952    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="storage-initializer"
Sep 11 16:45:35.609972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609961    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="storage-initializer"
Sep 11 16:45:35.609972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609966    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="storage-initializer"
Sep 11 16:45:35.609972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609971    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kube-rbac-proxy"
Sep 11 16:45:35.610263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609977    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kube-rbac-proxy"
Sep 11 16:45:35.610263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609989    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="kserve-container"
Sep 11 16:45:35.610263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.609994    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="kserve-container"
Sep 11 16:45:35.610263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.610038    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kube-rbac-proxy"
Sep 11 16:45:35.610263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.610047    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f348118-9023-47bb-baae-59061bc7b9da" containerName="kserve-container"
Sep 11 16:45:35.610263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.610054    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="kserve-container"
Sep 11 16:45:35.610263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.610059    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4871cb8-cee5-45e4-bfc9-b53b6680233a" containerName="kube-rbac-proxy"
Sep 11 16:45:35.614691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.614674    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.617446 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.617422    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 11 16:45:35.617569 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.617496    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:45:35.617569 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.617515    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:45:35.617569 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.617537    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:45:35.617569 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.617546    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:45:35.618445 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.618430    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 11 16:45:35.618528 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.618511    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:45:35.625146 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.625126    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"]
Sep 11 16:45:35.704740 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.704699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.704740 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.704743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.705027 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.704924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.705027 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.705012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpw9z\" (UniqueName: \"kubernetes.io/projected/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-api-access-vpw9z\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.705126 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.705073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.760814 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.760781    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:35.805865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.805775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.805865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.805820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpw9z\" (UniqueName: \"kubernetes.io/projected/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-api-access-vpw9z\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.805865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.805863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.806181 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.805892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.806181 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.805925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.806309 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.806290    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.806642 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.806619    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.806642 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.806637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.808613 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.808589    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.815429 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.815398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpw9z\" (UniqueName: \"kubernetes.io/projected/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-api-access-vpw9z\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:35.925374 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:35.925323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:36.197958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:36.197825    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"]
Sep 11 16:45:36.200405 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:45:36.200380    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2393be1_a28f_4b38_a75d_6c6ede3e9f04.slice/crio-71a973082fe691d6770905b8f4a4bc8bf6ff25900e047489c3eac5bf54535b30 WatchSource:0}: Error finding container 71a973082fe691d6770905b8f4a4bc8bf6ff25900e047489c3eac5bf54535b30: Status 404 returned error can't find the container with id 71a973082fe691d6770905b8f4a4bc8bf6ff25900e047489c3eac5bf54535b30
Sep 11 16:45:36.352399 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:36.352365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" event={"ID":"e2393be1-a28f-4b38-a75d-6c6ede3e9f04","Type":"ContainerStarted","Data":"cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da"}
Sep 11 16:45:36.352599 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:36.352423    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" event={"ID":"e2393be1-a28f-4b38-a75d-6c6ede3e9f04","Type":"ContainerStarted","Data":"71a973082fe691d6770905b8f4a4bc8bf6ff25900e047489c3eac5bf54535b30"}
Sep 11 16:45:42.181822 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:42.181788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:45.388333 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:45.388305    2576 generic.go:356] "Generic (PLEG): container finished" podID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerID="cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da" exitCode=0
Sep 11 16:45:45.388656 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:45.388377    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" event={"ID":"e2393be1-a28f-4b38-a75d-6c6ede3e9f04","Type":"ContainerDied","Data":"cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da"}
Sep 11 16:45:46.394134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:46.394102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" event={"ID":"e2393be1-a28f-4b38-a75d-6c6ede3e9f04","Type":"ContainerStarted","Data":"cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c"}
Sep 11 16:45:46.394518 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:46.394162    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" event={"ID":"e2393be1-a28f-4b38-a75d-6c6ede3e9f04","Type":"ContainerStarted","Data":"24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509"}
Sep 11 16:45:46.394518 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:46.394343    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:46.418801 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:46.418753    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" podStartSLOduration=2.312875781 podStartE2EDuration="11.418737153s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.105861372s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:45:46.417297801 +0000 UTC m=+1740.327098962" watchObservedRunningTime="2026-09-11 16:45:46.418737153 +0000 UTC m=+1740.328538314"
Sep 11 16:45:47.372262 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.372226    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:47.398879 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.398847    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:47.400478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.400435    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:45:47.419274 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.419240    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"]
Sep 11 16:45:47.422458 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.422441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.425640 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.425616    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 11 16:45:47.426108 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.426090    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 11 16:45:47.438097 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.438070    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"]
Sep 11 16:45:47.496159 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.496127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.496159 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.496162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.496408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.496260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q7hm\" (UniqueName: \"kubernetes.io/projected/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-api-access-7q7hm\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.496408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.496355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f75f3e63-f651-4ebc-b49e-12f472d247ff-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.496520 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.496460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f75f3e63-f651-4ebc-b49e-12f472d247ff-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.571317 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.571279    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:47.597180 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.597146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7hm\" (UniqueName: \"kubernetes.io/projected/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-api-access-7q7hm\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.597180 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.597181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f75f3e63-f651-4ebc-b49e-12f472d247ff-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.597384 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.597212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f75f3e63-f651-4ebc-b49e-12f472d247ff-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.597384 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.597354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.597465 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.597398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.597625 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.597601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f75f3e63-f651-4ebc-b49e-12f472d247ff-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.598052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.598036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.598133 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.598111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.599817 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.599797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f75f3e63-f651-4ebc-b49e-12f472d247ff-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.607759 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.607736    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7hm\" (UniqueName: \"kubernetes.io/projected/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-api-access-7q7hm\") pod \"isvc-sklearn-v2-predictor-ffc44684-v5xxc\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:47.732486 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:47.732442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:48.013427 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:48.013368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"]
Sep 11 16:45:48.015410 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:45:48.015381    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75f3e63_f651_4ebc_b49e_12f472d247ff.slice/crio-46cf0485738ddc72479680532ebefb0fae5528973dc88fe0cf047864a695389c WatchSource:0}: Error finding container 46cf0485738ddc72479680532ebefb0fae5528973dc88fe0cf047864a695389c: Status 404 returned error can't find the container with id 46cf0485738ddc72479680532ebefb0fae5528973dc88fe0cf047864a695389c
Sep 11 16:45:48.407993 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:48.407867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" event={"ID":"f75f3e63-f651-4ebc-b49e-12f472d247ff","Type":"ContainerStarted","Data":"eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6"}
Sep 11 16:45:48.407993 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:48.407976    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" event={"ID":"f75f3e63-f651-4ebc-b49e-12f472d247ff","Type":"ContainerStarted","Data":"46cf0485738ddc72479680532ebefb0fae5528973dc88fe0cf047864a695389c"}
Sep 11 16:45:48.408560 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:48.408531    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:45:52.424916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:52.424876    2576 generic.go:356] "Generic (PLEG): container finished" podID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerID="eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6" exitCode=0
Sep 11 16:45:52.425370 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:52.424924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" event={"ID":"f75f3e63-f651-4ebc-b49e-12f472d247ff","Type":"ContainerDied","Data":"eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6"}
Sep 11 16:45:53.412260 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:53.412229    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:45:53.412735 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:53.412710    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 11 16:45:53.431679 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:53.431648    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" event={"ID":"f75f3e63-f651-4ebc-b49e-12f472d247ff","Type":"ContainerStarted","Data":"d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6"}
Sep 11 16:45:53.432106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:53.431702    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" event={"ID":"f75f3e63-f651-4ebc-b49e-12f472d247ff","Type":"ContainerStarted","Data":"2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497"}
Sep 11 16:45:53.432106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:53.431981    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:53.432106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:53.432042    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:53.433361 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:53.433336    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:45:53.464227 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:53.464159    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podStartSLOduration=2.752382915 podStartE2EDuration="6.464138245s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.71175533s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:45:53.462852741 +0000 UTC m=+1747.372653904" watchObservedRunningTime="2026-09-11 16:45:53.464138245 +0000 UTC m=+1747.373939408"
Sep 11 16:45:54.436233 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:54.436195    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:45:59.440640 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:59.440614    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:45:59.441122 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:45:59.441098    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:46:03.413985 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:03.413924    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:46:09.441391 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:09.441350    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:46:18.143823 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:18.143791    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"]
Sep 11 16:46:18.144365 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:18.144111    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kserve-container" containerID="cri-o://24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509" gracePeriod=30
Sep 11 16:46:18.144365 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:18.144164    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kube-rbac-proxy" containerID="cri-o://cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c" gracePeriod=30
Sep 11 16:46:18.176203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:18.176174    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:18.408604 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:18.408509    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 11 16:46:18.536592 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:18.536559    2576 generic.go:356] "Generic (PLEG): container finished" podID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerID="cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c" exitCode=2
Sep 11 16:46:18.536765 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:18.536635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" event={"ID":"e2393be1-a28f-4b38-a75d-6c6ede3e9f04","Type":"ContainerDied","Data":"cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c"}
Sep 11 16:46:19.441843 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.441809    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:46:19.803966 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.803928    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:46:19.969616 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.969535    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-rbac-proxy-sar-config\") pod \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") "
Sep 11 16:46:19.969616 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.969575    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kserve-provision-location\") pod \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") "
Sep 11 16:46:19.969616 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.969601    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-proxy-tls\") pod \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") "
Sep 11 16:46:19.969616 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.969616    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-cabundle-cert\") pod \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") "
Sep 11 16:46:19.969877 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.969738    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpw9z\" (UniqueName: \"kubernetes.io/projected/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-api-access-vpw9z\") pod \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\" (UID: \"e2393be1-a28f-4b38-a75d-6c6ede3e9f04\") "
Sep 11 16:46:19.970009 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.969979    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2393be1-a28f-4b38-a75d-6c6ede3e9f04" (UID: "e2393be1-a28f-4b38-a75d-6c6ede3e9f04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:46:19.970009 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.969992    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2393be1-a28f-4b38-a75d-6c6ede3e9f04" (UID: "e2393be1-a28f-4b38-a75d-6c6ede3e9f04"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:46:19.970143 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.970058    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:46:19.970143 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.970078    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:46:19.970656 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.970634    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2393be1-a28f-4b38-a75d-6c6ede3e9f04" (UID: "e2393be1-a28f-4b38-a75d-6c6ede3e9f04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:19.971865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.971842    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-api-access-vpw9z" (OuterVolumeSpecName: "kube-api-access-vpw9z") pod "e2393be1-a28f-4b38-a75d-6c6ede3e9f04" (UID: "e2393be1-a28f-4b38-a75d-6c6ede3e9f04"). InnerVolumeSpecName "kube-api-access-vpw9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:46:19.971982 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:19.971892    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2393be1-a28f-4b38-a75d-6c6ede3e9f04" (UID: "e2393be1-a28f-4b38-a75d-6c6ede3e9f04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:46:20.071203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.071153    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpw9z\" (UniqueName: \"kubernetes.io/projected/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kube-api-access-vpw9z\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:46:20.071203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.071196    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:46:20.071203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.071206    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2393be1-a28f-4b38-a75d-6c6ede3e9f04-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:46:20.547763 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.547728    2576 generic.go:356] "Generic (PLEG): container finished" podID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerID="24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509" exitCode=0
Sep 11 16:46:20.548197 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.547790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" event={"ID":"e2393be1-a28f-4b38-a75d-6c6ede3e9f04","Type":"ContainerDied","Data":"24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509"}
Sep 11 16:46:20.548197 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.547858    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w" event={"ID":"e2393be1-a28f-4b38-a75d-6c6ede3e9f04","Type":"ContainerDied","Data":"71a973082fe691d6770905b8f4a4bc8bf6ff25900e047489c3eac5bf54535b30"}
Sep 11 16:46:20.548197 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.547792    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"
Sep 11 16:46:20.548197 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.547883    2576 scope.go:117] "RemoveContainer" containerID="cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c"
Sep 11 16:46:20.556541 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.556525    2576 scope.go:117] "RemoveContainer" containerID="24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509"
Sep 11 16:46:20.564105 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.564090    2576 scope.go:117] "RemoveContainer" containerID="cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da"
Sep 11 16:46:20.570916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.570898    2576 scope.go:117] "RemoveContainer" containerID="cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c"
Sep 11 16:46:20.571210 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:46:20.571191    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c\": container with ID starting with cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c not found: ID does not exist" containerID="cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c"
Sep 11 16:46:20.571259 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.571219    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c"} err="failed to get container status \"cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c\": rpc error: code = NotFound desc = could not find container \"cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c\": container with ID starting with cb0dc8c0da5da91078eb1c622956264559d88d06ffed7265b1dbd11adbfbbe1c not found: ID does not exist"
Sep 11 16:46:20.571259 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.571237    2576 scope.go:117] "RemoveContainer" containerID="24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509"
Sep 11 16:46:20.571508 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:46:20.571490    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509\": container with ID starting with 24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509 not found: ID does not exist" containerID="24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509"
Sep 11 16:46:20.571565 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.571512    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509"} err="failed to get container status \"24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509\": rpc error: code = NotFound desc = could not find container \"24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509\": container with ID starting with 24857a2097c066c953e8e45e1f3febc2067d4fa80b6aedd708f66a22d606b509 not found: ID does not exist"
Sep 11 16:46:20.571565 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.571528    2576 scope.go:117] "RemoveContainer" containerID="cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da"
Sep 11 16:46:20.571754 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:46:20.571739    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da\": container with ID starting with cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da not found: ID does not exist" containerID="cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da"
Sep 11 16:46:20.571801 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.571757    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da"} err="failed to get container status \"cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da\": rpc error: code = NotFound desc = could not find container \"cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da\": container with ID starting with cb11dba0770fc07a51ae9110ef0b162dd3850497670ecf2d342f1ef15f3817da not found: ID does not exist"
Sep 11 16:46:20.581571 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.581551    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"]
Sep 11 16:46:20.586128 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.586106    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w"]
Sep 11 16:46:20.603102 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:20.603081    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" path="/var/lib/kubelet/pods/e2393be1-a28f-4b38-a75d-6c6ede3e9f04/volumes"
Sep 11 16:46:23.310042 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.310014    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:23.388822 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.388780    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"]
Sep 11 16:46:23.389179 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.389161    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kserve-container"
Sep 11 16:46:23.389272 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.389182    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kserve-container"
Sep 11 16:46:23.389272 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.389207    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kube-rbac-proxy"
Sep 11 16:46:23.389272 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.389216    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kube-rbac-proxy"
Sep 11 16:46:23.389272 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.389231    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="storage-initializer"
Sep 11 16:46:23.389272 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.389239    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="storage-initializer"
Sep 11 16:46:23.389536 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.389322    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kserve-container"
Sep 11 16:46:23.389536 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.389337    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2393be1-a28f-4b38-a75d-6c6ede3e9f04" containerName="kube-rbac-proxy"
Sep 11 16:46:23.394367 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.394342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.397743 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.397723    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 11 16:46:23.398045 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.398028    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 11 16:46:23.407351 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.407330    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"]
Sep 11 16:46:23.498834 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.498795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.499054 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.498844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.499054 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.498866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.499054 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.499001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.499054 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.499042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqpfs\" (UniqueName: \"kubernetes.io/projected/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-api-access-jqpfs\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.540197 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.540167    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:23.600294 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.600205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqpfs\" (UniqueName: \"kubernetes.io/projected/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-api-access-jqpfs\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.600294 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.600265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.600501 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.600298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.600501 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.600322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.600501 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.600364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.600758 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.600737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.601126 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.601103    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.601220 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.601121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.602893 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.602873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.610396 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.610376    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqpfs\" (UniqueName: \"kubernetes.io/projected/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-api-access-jqpfs\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.703972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.703919    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:23.981974 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:23.981928    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"]
Sep 11 16:46:23.982978 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:46:23.982952    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda298fc64_cf88_4b85_b9c6_d04a4b8a6564.slice/crio-9445af7d2d8c77cece9888f1fa70df327d17b5148fa526dea02717d0a5d2f16a WatchSource:0}: Error finding container 9445af7d2d8c77cece9888f1fa70df327d17b5148fa526dea02717d0a5d2f16a: Status 404 returned error can't find the container with id 9445af7d2d8c77cece9888f1fa70df327d17b5148fa526dea02717d0a5d2f16a
Sep 11 16:46:24.565785 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:24.565748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" event={"ID":"a298fc64-cf88-4b85-b9c6-d04a4b8a6564","Type":"ContainerStarted","Data":"217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69"}
Sep 11 16:46:24.566204 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:24.565822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" event={"ID":"a298fc64-cf88-4b85-b9c6-d04a4b8a6564","Type":"ContainerStarted","Data":"9445af7d2d8c77cece9888f1fa70df327d17b5148fa526dea02717d0a5d2f16a"}
Sep 11 16:46:28.582066 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:28.582031    2576 generic.go:356] "Generic (PLEG): container finished" podID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerID="217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69" exitCode=0
Sep 11 16:46:28.582387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:28.582082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" event={"ID":"a298fc64-cf88-4b85-b9c6-d04a4b8a6564","Type":"ContainerDied","Data":"217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69"}
Sep 11 16:46:29.441548 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:29.441510    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:46:29.588769 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:29.588728    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" event={"ID":"a298fc64-cf88-4b85-b9c6-d04a4b8a6564","Type":"ContainerStarted","Data":"8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7"}
Sep 11 16:46:29.589282 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:29.588810    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" event={"ID":"a298fc64-cf88-4b85-b9c6-d04a4b8a6564","Type":"ContainerStarted","Data":"c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7"}
Sep 11 16:46:29.589282 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:29.589050    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:29.621896 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:29.621827    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" podStartSLOduration=2.120791995 podStartE2EDuration="6.621806195s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.5010142s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:46:29.621697079 +0000 UTC m=+1783.531498241" watchObservedRunningTime="2026-09-11 16:46:29.621806195 +0000 UTC m=+1783.531607384"
Sep 11 16:46:30.592553 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:30.592521    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:30.593988 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:30.593954    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:46:31.596845 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:31.596795    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:46:36.602903 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:36.602876    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:36.603379 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:36.603314    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 11 16:46:39.441296 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:39.441204    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:46:46.578735 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:46.578705    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:46:46.581107 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:46.581084    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:46:46.588349 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:46.588327    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:46:46.592481 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:46.592462    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:46:46.603990 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:46.603971    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:46:49.441066 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:49.441023    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:46:58.509248 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:58.509216    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"]
Sep 11 16:46:58.509694 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:58.509490    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kserve-container" containerID="cri-o://c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7" gracePeriod=30
Sep 11 16:46:58.509694 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:58.509500    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kube-rbac-proxy" containerID="cri-o://8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7" gracePeriod=30
Sep 11 16:46:58.516048 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:58.516024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:58.707718 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:58.707685    2576 generic.go:356] "Generic (PLEG): container finished" podID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerID="8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7" exitCode=2
Sep 11 16:46:58.707899 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:58.707754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" event={"ID":"a298fc64-cf88-4b85-b9c6-d04a4b8a6564","Type":"ContainerDied","Data":"8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7"}
Sep 11 16:46:59.441946 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:59.441891    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:46:59.958111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:46:59.958084    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:47:00.084168 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.084068    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kserve-provision-location\") pod \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") "
Sep 11 16:47:00.084168 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.084130    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-proxy-tls\") pod \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") "
Sep 11 16:47:00.084168 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.084151    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-rbac-proxy-sar-config\") pod \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") "
Sep 11 16:47:00.084168 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.084171    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqpfs\" (UniqueName: \"kubernetes.io/projected/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-api-access-jqpfs\") pod \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") "
Sep 11 16:47:00.084518 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.084207    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-cabundle-cert\") pod \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\" (UID: \"a298fc64-cf88-4b85-b9c6-d04a4b8a6564\") "
Sep 11 16:47:00.084581 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.084521    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a298fc64-cf88-4b85-b9c6-d04a4b8a6564" (UID: "a298fc64-cf88-4b85-b9c6-d04a4b8a6564"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:47:00.084581 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.084566    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a298fc64-cf88-4b85-b9c6-d04a4b8a6564" (UID: "a298fc64-cf88-4b85-b9c6-d04a4b8a6564"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:47:00.085110 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.085086    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a298fc64-cf88-4b85-b9c6-d04a4b8a6564" (UID: "a298fc64-cf88-4b85-b9c6-d04a4b8a6564"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:47:00.086474 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.086443    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-api-access-jqpfs" (OuterVolumeSpecName: "kube-api-access-jqpfs") pod "a298fc64-cf88-4b85-b9c6-d04a4b8a6564" (UID: "a298fc64-cf88-4b85-b9c6-d04a4b8a6564"). InnerVolumeSpecName "kube-api-access-jqpfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:47:00.086568 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.086501    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a298fc64-cf88-4b85-b9c6-d04a4b8a6564" (UID: "a298fc64-cf88-4b85-b9c6-d04a4b8a6564"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:47:00.185069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.185024    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqpfs\" (UniqueName: \"kubernetes.io/projected/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-api-access-jqpfs\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:00.185069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.185056    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:00.185069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.185065    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:00.185069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.185079    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:00.185354 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.185087    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a298fc64-cf88-4b85-b9c6-d04a4b8a6564-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:00.717830 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.717788    2576 generic.go:356] "Generic (PLEG): container finished" podID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerID="c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7" exitCode=0
Sep 11 16:47:00.718021 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.717847    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"
Sep 11 16:47:00.718021 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.717863    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" event={"ID":"a298fc64-cf88-4b85-b9c6-d04a4b8a6564","Type":"ContainerDied","Data":"c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7"}
Sep 11 16:47:00.718021 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.717911    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5" event={"ID":"a298fc64-cf88-4b85-b9c6-d04a4b8a6564","Type":"ContainerDied","Data":"9445af7d2d8c77cece9888f1fa70df327d17b5148fa526dea02717d0a5d2f16a"}
Sep 11 16:47:00.718021 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.717954    2576 scope.go:117] "RemoveContainer" containerID="8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7"
Sep 11 16:47:00.726119 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.726096    2576 scope.go:117] "RemoveContainer" containerID="c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7"
Sep 11 16:47:00.733686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.733667    2576 scope.go:117] "RemoveContainer" containerID="217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69"
Sep 11 16:47:00.740518 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.740502    2576 scope.go:117] "RemoveContainer" containerID="8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7"
Sep 11 16:47:00.740767 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:47:00.740747    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7\": container with ID starting with 8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7 not found: ID does not exist" containerID="8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7"
Sep 11 16:47:00.740829 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.740775    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7"} err="failed to get container status \"8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7\": rpc error: code = NotFound desc = could not find container \"8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7\": container with ID starting with 8dd9158d61ce557282f446b288db5fa66fb2b68b8a6541cacd6e195a544f74f7 not found: ID does not exist"
Sep 11 16:47:00.740829 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.740790    2576 scope.go:117] "RemoveContainer" containerID="c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7"
Sep 11 16:47:00.741037 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:47:00.741022    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7\": container with ID starting with c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7 not found: ID does not exist" containerID="c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7"
Sep 11 16:47:00.741083 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.741042    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7"} err="failed to get container status \"c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7\": rpc error: code = NotFound desc = could not find container \"c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7\": container with ID starting with c1d52b0a9834d895a516946545aa6b999300264684223c634d1cdd92f10ce2c7 not found: ID does not exist"
Sep 11 16:47:00.741083 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.741057    2576 scope.go:117] "RemoveContainer" containerID="217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69"
Sep 11 16:47:00.741308 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:47:00.741290    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69\": container with ID starting with 217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69 not found: ID does not exist" containerID="217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69"
Sep 11 16:47:00.741357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.741314    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69"} err="failed to get container status \"217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69\": rpc error: code = NotFound desc = could not find container \"217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69\": container with ID starting with 217563c900d7b6e79df5f57ed8a6aa16b3622bbda1ae6eca9ba31266621f8f69 not found: ID does not exist"
Sep 11 16:47:00.747116 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.747087    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"]
Sep 11 16:47:00.753450 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:00.753428    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5"]
Sep 11 16:47:02.603010 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:02.602979    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" path="/var/lib/kubelet/pods/a298fc64-cf88-4b85-b9c6-d04a4b8a6564/volumes"
Sep 11 16:47:03.730006 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.729975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:03.800919 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.800877    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"]
Sep 11 16:47:03.801195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.801183    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kube-rbac-proxy"
Sep 11 16:47:03.801251 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.801197    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kube-rbac-proxy"
Sep 11 16:47:03.801251 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.801218    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="storage-initializer"
Sep 11 16:47:03.801251 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.801223    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="storage-initializer"
Sep 11 16:47:03.801251 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.801231    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kserve-container"
Sep 11 16:47:03.801251 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.801237    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kserve-container"
Sep 11 16:47:03.801401 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.801283    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kserve-container"
Sep 11 16:47:03.801401 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.801294    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a298fc64-cf88-4b85-b9c6-d04a4b8a6564" containerName="kube-rbac-proxy"
Sep 11 16:47:03.806043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.806024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:03.811755 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.811703    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 11 16:47:03.812411 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.812110    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:03.813346 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.813318    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 11 16:47:03.822655 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.822631    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"]
Sep 11 16:47:03.881848 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.881815    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"]
Sep 11 16:47:03.885215 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.885194    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:03.888523 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.888495    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 11 16:47:03.899749 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.899716    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"]
Sep 11 16:47:03.900730 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.900704    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:03.916540 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.916513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:03.916711 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.916544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:03.916711 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.916567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:03.916711 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.916634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:03.916711 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:03.916705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf5zh\" (UniqueName: \"kubernetes.io/projected/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-api-access-wf5zh\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.017525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.017525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf5zh\" (UniqueName: \"kubernetes.io/projected/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-api-access-wf5zh\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.017525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.017811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.017811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.017811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.017811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.018064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.018064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.018064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.017906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wpg\" (UniqueName: \"kubernetes.io/projected/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-api-access-p2wpg\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.018325 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.018301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.018515 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.018497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.018643 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.018622    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.020347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.020323    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.031894 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.031868    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf5zh\" (UniqueName: \"kubernetes.io/projected/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-api-access-wf5zh\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-v25rb\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.115738 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.115703    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:04.116646 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.116624    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:04.118957 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.118900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.119074 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.118959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wpg\" (UniqueName: \"kubernetes.io/projected/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-api-access-p2wpg\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.119074 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.118990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.119074 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.119028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.119074 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.119070    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.119500 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.119473    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.119713 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.119691    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.120085 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.119698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.121808 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.121787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.131104 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.131075    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wpg\" (UniqueName: \"kubernetes.io/projected/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-api-access-p2wpg\") pod \"isvc-canary-create-predictor-6474b875b9-l68gc\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.198287 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.198255    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:04.397707 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.397626    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"]
Sep 11 16:47:04.400727 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:47:04.400697    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047b51d0_01e5_40b7_b5b7_2dec5c50aa03.slice/crio-6f36a4944b5e4630d43a44a3775bbdc9978a267d7d4b6be1ecff53b2e31b212e WatchSource:0}: Error finding container 6f36a4944b5e4630d43a44a3775bbdc9978a267d7d4b6be1ecff53b2e31b212e: Status 404 returned error can't find the container with id 6f36a4944b5e4630d43a44a3775bbdc9978a267d7d4b6be1ecff53b2e31b212e
Sep 11 16:47:04.477915 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.477885    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"]
Sep 11 16:47:04.479804 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:47:04.479778    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dd3d30_20fa_4570_95b2_d4bcec00cb5e.slice/crio-11aebd3cc72e41334e31b6e1d5b3a2c037408522c88c92a7fc8893fd6d1b6d27 WatchSource:0}: Error finding container 11aebd3cc72e41334e31b6e1d5b3a2c037408522c88c92a7fc8893fd6d1b6d27: Status 404 returned error can't find the container with id 11aebd3cc72e41334e31b6e1d5b3a2c037408522c88c92a7fc8893fd6d1b6d27
Sep 11 16:47:04.736540 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.736503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" event={"ID":"047b51d0-01e5-40b7-b5b7-2dec5c50aa03","Type":"ContainerStarted","Data":"2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869"}
Sep 11 16:47:04.737103 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.736576    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" event={"ID":"047b51d0-01e5-40b7-b5b7-2dec5c50aa03","Type":"ContainerStarted","Data":"6f36a4944b5e4630d43a44a3775bbdc9978a267d7d4b6be1ecff53b2e31b212e"}
Sep 11 16:47:04.738428 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.738380    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" event={"ID":"89dd3d30-20fa-4570-95b2-d4bcec00cb5e","Type":"ContainerStarted","Data":"5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55"}
Sep 11 16:47:04.738550 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:04.738451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" event={"ID":"89dd3d30-20fa-4570-95b2-d4bcec00cb5e","Type":"ContainerStarted","Data":"11aebd3cc72e41334e31b6e1d5b3a2c037408522c88c92a7fc8893fd6d1b6d27"}
Sep 11 16:47:06.599760 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:06.599717    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 11 16:47:08.755118 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:08.755078    2576 generic.go:356] "Generic (PLEG): container finished" podID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerID="2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869" exitCode=0
Sep 11 16:47:08.755552 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:08.755151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" event={"ID":"047b51d0-01e5-40b7-b5b7-2dec5c50aa03","Type":"ContainerDied","Data":"2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869"}
Sep 11 16:47:08.756794 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:08.756769    2576 generic.go:356] "Generic (PLEG): container finished" podID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerID="5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55" exitCode=0
Sep 11 16:47:08.756883 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:08.756804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" event={"ID":"89dd3d30-20fa-4570-95b2-d4bcec00cb5e","Type":"ContainerDied","Data":"5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55"}
Sep 11 16:47:09.762981 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.762927    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" event={"ID":"047b51d0-01e5-40b7-b5b7-2dec5c50aa03","Type":"ContainerStarted","Data":"c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42"}
Sep 11 16:47:09.763470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.763016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" event={"ID":"047b51d0-01e5-40b7-b5b7-2dec5c50aa03","Type":"ContainerStarted","Data":"153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f"}
Sep 11 16:47:09.763470 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.763253    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:09.764984 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.764962    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" event={"ID":"89dd3d30-20fa-4570-95b2-d4bcec00cb5e","Type":"ContainerStarted","Data":"22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a"}
Sep 11 16:47:09.765083 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.765010    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" event={"ID":"89dd3d30-20fa-4570-95b2-d4bcec00cb5e","Type":"ContainerStarted","Data":"046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd"}
Sep 11 16:47:09.765634 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.765251    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:09.765818 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.765805    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:09.770543 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.766701    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:47:09.814191 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.814137    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podStartSLOduration=3.028049703 podStartE2EDuration="6.814122181s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786072478s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:47:09.814072232 +0000 UTC m=+1823.723873406" watchObservedRunningTime="2026-09-11 16:47:09.814122181 +0000 UTC m=+1823.723923343"
Sep 11 16:47:09.858971 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:09.858899    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podStartSLOduration=2.985122416 podStartE2EDuration="6.858884189s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.873761773s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:47:09.857982594 +0000 UTC m=+1823.767783756" watchObservedRunningTime="2026-09-11 16:47:09.858884189 +0000 UTC m=+1823.768685350"
Sep 11 16:47:10.769145 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:10.769093    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:47:10.769145 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:10.769137    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:10.770145 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:10.770123    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:47:11.773131 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:11.773092    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:47:15.773669 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:15.773638    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:47:15.774222 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:15.774152    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:47:16.603710 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:16.603681    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:47:16.777316 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:16.777287    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:47:16.777883 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:16.777855    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:47:17.692739 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:17.692704    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"]
Sep 11 16:47:17.693002 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:17.692976    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container" containerID="cri-o://2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497" gracePeriod=30
Sep 11 16:47:17.693160 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:17.693015    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kube-rbac-proxy" containerID="cri-o://d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6" gracePeriod=30
Sep 11 16:47:17.706158 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:17.706133    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:18.803779 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:18.803749    2576 generic.go:356] "Generic (PLEG): container finished" podID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerID="d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6" exitCode=2
Sep 11 16:47:18.804171 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:18.803815    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" event={"ID":"f75f3e63-f651-4ebc-b49e-12f472d247ff","Type":"ContainerDied","Data":"d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6"}
Sep 11 16:47:19.437041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:19.437001    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 11 16:47:22.741824 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.741801    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:47:22.755810 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.755727    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:22.821509 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.821479    2576 generic.go:356] "Generic (PLEG): container finished" podID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerID="2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497" exitCode=0
Sep 11 16:47:22.821659 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.821539    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"
Sep 11 16:47:22.821659 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.821548    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" event={"ID":"f75f3e63-f651-4ebc-b49e-12f472d247ff","Type":"ContainerDied","Data":"2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497"}
Sep 11 16:47:22.821659 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.821595    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc" event={"ID":"f75f3e63-f651-4ebc-b49e-12f472d247ff","Type":"ContainerDied","Data":"46cf0485738ddc72479680532ebefb0fae5528973dc88fe0cf047864a695389c"}
Sep 11 16:47:22.821659 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.821625    2576 scope.go:117] "RemoveContainer" containerID="d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6"
Sep 11 16:47:22.829455 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.829435    2576 scope.go:117] "RemoveContainer" containerID="2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497"
Sep 11 16:47:22.837341 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.837318    2576 scope.go:117] "RemoveContainer" containerID="eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6"
Sep 11 16:47:22.844641 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.844626    2576 scope.go:117] "RemoveContainer" containerID="d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6"
Sep 11 16:47:22.845083 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:47:22.845032    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6\": container with ID starting with d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6 not found: ID does not exist" containerID="d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6"
Sep 11 16:47:22.845205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.845081    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6"} err="failed to get container status \"d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6\": rpc error: code = NotFound desc = could not find container \"d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6\": container with ID starting with d72bd9fde51235eb37ab728c3e406d22208db746dbbf9f2409f73fbb14aafcc6 not found: ID does not exist"
Sep 11 16:47:22.845205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.845108    2576 scope.go:117] "RemoveContainer" containerID="2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497"
Sep 11 16:47:22.845504 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:47:22.845481    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497\": container with ID starting with 2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497 not found: ID does not exist" containerID="2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497"
Sep 11 16:47:22.845635 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.845543    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497"} err="failed to get container status \"2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497\": rpc error: code = NotFound desc = could not find container \"2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497\": container with ID starting with 2ee8190bdac5ba7455299a8a7fe8036ae15119773467a75755351c643e15e497 not found: ID does not exist"
Sep 11 16:47:22.845635 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.845593    2576 scope.go:117] "RemoveContainer" containerID="eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6"
Sep 11 16:47:22.845955 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:47:22.845903    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6\": container with ID starting with eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6 not found: ID does not exist" containerID="eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6"
Sep 11 16:47:22.846061 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.845959    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6"} err="failed to get container status \"eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6\": rpc error: code = NotFound desc = could not find container \"eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6\": container with ID starting with eea488de53622e372b44ceffa8009b4960122f3b76199ca8508194b2061fa5e6 not found: ID does not exist"
Sep 11 16:47:22.846912 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.846895    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"]
Sep 11 16:47:22.847379 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.847362    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container"
Sep 11 16:47:22.847436 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.847383    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container"
Sep 11 16:47:22.847436 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.847414    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kube-rbac-proxy"
Sep 11 16:47:22.847436 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.847423    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kube-rbac-proxy"
Sep 11 16:47:22.847549 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.847439    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="storage-initializer"
Sep 11 16:47:22.847549 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.847449    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="storage-initializer"
Sep 11 16:47:22.847549 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.847546    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kserve-container"
Sep 11 16:47:22.847655 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.847560    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" containerName="kube-rbac-proxy"
Sep 11 16:47:22.852110 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.852095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:22.855244 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.855221    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 11 16:47:22.855323 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.855266    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 11 16:47:22.873429 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.873404    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"]
Sep 11 16:47:22.876572 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.876550    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-cabundle-cert\") pod \"f75f3e63-f651-4ebc-b49e-12f472d247ff\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") "
Sep 11 16:47:22.876667 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.876590    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f75f3e63-f651-4ebc-b49e-12f472d247ff-proxy-tls\") pod \"f75f3e63-f651-4ebc-b49e-12f472d247ff\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") "
Sep 11 16:47:22.876667 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.876624    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f75f3e63-f651-4ebc-b49e-12f472d247ff-kserve-provision-location\") pod \"f75f3e63-f651-4ebc-b49e-12f472d247ff\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") "
Sep 11 16:47:22.876759 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.876669    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q7hm\" (UniqueName: \"kubernetes.io/projected/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-api-access-7q7hm\") pod \"f75f3e63-f651-4ebc-b49e-12f472d247ff\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") "
Sep 11 16:47:22.876759 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.876693    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-rbac-proxy-sar-config\") pod \"f75f3e63-f651-4ebc-b49e-12f472d247ff\" (UID: \"f75f3e63-f651-4ebc-b49e-12f472d247ff\") "
Sep 11 16:47:22.876971 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.876918    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f75f3e63-f651-4ebc-b49e-12f472d247ff" (UID: "f75f3e63-f651-4ebc-b49e-12f472d247ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:47:22.877071 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.876999    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75f3e63-f651-4ebc-b49e-12f472d247ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f75f3e63-f651-4ebc-b49e-12f472d247ff" (UID: "f75f3e63-f651-4ebc-b49e-12f472d247ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:47:22.877160 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.877134    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f75f3e63-f651-4ebc-b49e-12f472d247ff" (UID: "f75f3e63-f651-4ebc-b49e-12f472d247ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:47:22.878877 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.878856    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-api-access-7q7hm" (OuterVolumeSpecName: "kube-api-access-7q7hm") pod "f75f3e63-f651-4ebc-b49e-12f472d247ff" (UID: "f75f3e63-f651-4ebc-b49e-12f472d247ff"). InnerVolumeSpecName "kube-api-access-7q7hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:47:22.878982 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.878889    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f75f3e63-f651-4ebc-b49e-12f472d247ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f75f3e63-f651-4ebc-b49e-12f472d247ff" (UID: "f75f3e63-f651-4ebc-b49e-12f472d247ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:47:22.977660 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtw6x\" (UniqueName: \"kubernetes.io/projected/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-api-access-xtw6x\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:22.977854 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:22.977854 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:22.977854 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:22.977854 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:22.978052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977900    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f75f3e63-f651-4ebc-b49e-12f472d247ff-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:22.978052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977916    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q7hm\" (UniqueName: \"kubernetes.io/projected/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-api-access-7q7hm\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:22.978052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977925    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:22.978052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977956    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f75f3e63-f651-4ebc-b49e-12f472d247ff-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:22.978052 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:22.977966    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f75f3e63-f651-4ebc-b49e-12f472d247ff-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:47:23.055914 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.055838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:23.078916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.078885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.079111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.078921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.079111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.078977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.079111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.079000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.079307 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.079223    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtw6x\" (UniqueName: \"kubernetes.io/projected/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-api-access-xtw6x\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.079444 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.079425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.079641 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.079621    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.079706 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.079648    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.081612 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.081594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.097082 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.097056    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtw6x\" (UniqueName: \"kubernetes.io/projected/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-api-access-xtw6x\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.150691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.150663    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"]
Sep 11 16:47:23.153366 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.153342    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc"]
Sep 11 16:47:23.162211 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.162192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:23.444523 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.444499    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"]
Sep 11 16:47:23.445872 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:47:23.445844    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71527f2c_27fe_4c7f_8a6f_aa89c897dc52.slice/crio-c0192f75e273ced377c16561e79f78f7f67ff2de99e184caee6d65a505883f56 WatchSource:0}: Error finding container c0192f75e273ced377c16561e79f78f7f67ff2de99e184caee6d65a505883f56: Status 404 returned error can't find the container with id c0192f75e273ced377c16561e79f78f7f67ff2de99e184caee6d65a505883f56
Sep 11 16:47:23.828513 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.828432    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" event={"ID":"71527f2c-27fe-4c7f-8a6f-aa89c897dc52","Type":"ContainerStarted","Data":"41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075"}
Sep 11 16:47:23.828513 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:23.828488    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" event={"ID":"71527f2c-27fe-4c7f-8a6f-aa89c897dc52","Type":"ContainerStarted","Data":"c0192f75e273ced377c16561e79f78f7f67ff2de99e184caee6d65a505883f56"}
Sep 11 16:47:24.603188 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:24.603155    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f75f3e63-f651-4ebc-b49e-12f472d247ff" path="/var/lib/kubelet/pods/f75f3e63-f651-4ebc-b49e-12f472d247ff/volumes"
Sep 11 16:47:25.774565 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:25.774525    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:47:26.778002 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:26.777962    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:47:27.848997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:27.848877    2576 generic.go:356] "Generic (PLEG): container finished" podID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerID="41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075" exitCode=0
Sep 11 16:47:27.848997 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:27.848917    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" event={"ID":"71527f2c-27fe-4c7f-8a6f-aa89c897dc52","Type":"ContainerDied","Data":"41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075"}
Sep 11 16:47:28.855513 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:28.855477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" event={"ID":"71527f2c-27fe-4c7f-8a6f-aa89c897dc52","Type":"ContainerStarted","Data":"07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431"}
Sep 11 16:47:28.855968 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:28.855544    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" event={"ID":"71527f2c-27fe-4c7f-8a6f-aa89c897dc52","Type":"ContainerStarted","Data":"7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59"}
Sep 11 16:47:28.855968 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:28.855819    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:28.855968 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:28.855927    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:28.857265 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:28.857240    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:47:29.860409 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:29.860370    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:47:34.865168 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:34.865137    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:47:34.865752 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:34.865725    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:47:34.890073 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:34.890029    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podStartSLOduration=8.86432851 podStartE2EDuration="12.890010715s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.025682205s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:47:28.903908693 +0000 UTC m=+1842.813709855" watchObservedRunningTime="2026-09-11 16:47:34.890010715 +0000 UTC m=+1848.799811877"
Sep 11 16:47:35.775091 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:35.775044    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:47:36.778576 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:36.778536    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:47:44.866313 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:44.866271    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:47:45.774270 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:45.774227    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:47:46.778657 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:46.778616    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:47:54.865862 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:54.865823    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:47:55.774531 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:55.774495    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:47:56.778546 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:47:56.778505    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:48:04.865713 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:04.865668    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:48:05.774481 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:05.774441    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:48:06.778009 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:06.777965    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:48:14.866542 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:14.866460    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:48:15.774730 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:15.774693    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:48:16.778351 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:16.778316    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:48:20.598096 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:20.598046    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 11 16:48:24.866191 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:24.866145    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:48:26.778465 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:26.778429    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:48:26.955524 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:26.955487    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"]
Sep 11 16:48:26.955777 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:26.955752    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container" containerID="cri-o://046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd" gracePeriod=30
Sep 11 16:48:26.955875 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:26.955818    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kube-rbac-proxy" containerID="cri-o://22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a" gracePeriod=30
Sep 11 16:48:26.956738 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:26.956718    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:48:27.107696 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:27.107665    2576 generic.go:356] "Generic (PLEG): container finished" podID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerID="22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a" exitCode=2
Sep 11 16:48:27.107856 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:27.107703    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" event={"ID":"89dd3d30-20fa-4570-95b2-d4bcec00cb5e","Type":"ContainerDied","Data":"22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a"}
Sep 11 16:48:30.769125 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:30.769076    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 11 16:48:31.799113 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.799082    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:48:31.928868 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.928780    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kserve-provision-location\") pod \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") "
Sep 11 16:48:31.928868 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.928827    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-rbac-proxy-sar-config\") pod \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") "
Sep 11 16:48:31.929127 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.928889    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2wpg\" (UniqueName: \"kubernetes.io/projected/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-api-access-p2wpg\") pod \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") "
Sep 11 16:48:31.929127 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.928927    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-cabundle-cert\") pod \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") "
Sep 11 16:48:31.929127 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.928983    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-proxy-tls\") pod \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\" (UID: \"89dd3d30-20fa-4570-95b2-d4bcec00cb5e\") "
Sep 11 16:48:31.929300 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.929184    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89dd3d30-20fa-4570-95b2-d4bcec00cb5e" (UID: "89dd3d30-20fa-4570-95b2-d4bcec00cb5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:48:31.929349 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.929291    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89dd3d30-20fa-4570-95b2-d4bcec00cb5e" (UID: "89dd3d30-20fa-4570-95b2-d4bcec00cb5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:48:31.929386 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.929343    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89dd3d30-20fa-4570-95b2-d4bcec00cb5e" (UID: "89dd3d30-20fa-4570-95b2-d4bcec00cb5e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:48:31.931228 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.931202    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89dd3d30-20fa-4570-95b2-d4bcec00cb5e" (UID: "89dd3d30-20fa-4570-95b2-d4bcec00cb5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:48:31.931325 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:31.931240    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-api-access-p2wpg" (OuterVolumeSpecName: "kube-api-access-p2wpg") pod "89dd3d30-20fa-4570-95b2-d4bcec00cb5e" (UID: "89dd3d30-20fa-4570-95b2-d4bcec00cb5e"). InnerVolumeSpecName "kube-api-access-p2wpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:48:32.029897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.029863    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2wpg\" (UniqueName: \"kubernetes.io/projected/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-api-access-p2wpg\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:32.029897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.029889    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:32.029897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.029899    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:32.029897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.029907    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:32.030173 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.029915    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89dd3d30-20fa-4570-95b2-d4bcec00cb5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:32.128534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.128499    2576 generic.go:356] "Generic (PLEG): container finished" podID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerID="046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd" exitCode=0
Sep 11 16:48:32.128693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.128554    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"
Sep 11 16:48:32.128693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.128582    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" event={"ID":"89dd3d30-20fa-4570-95b2-d4bcec00cb5e","Type":"ContainerDied","Data":"046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd"}
Sep 11 16:48:32.128693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.128640    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc" event={"ID":"89dd3d30-20fa-4570-95b2-d4bcec00cb5e","Type":"ContainerDied","Data":"11aebd3cc72e41334e31b6e1d5b3a2c037408522c88c92a7fc8893fd6d1b6d27"}
Sep 11 16:48:32.128693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.128667    2576 scope.go:117] "RemoveContainer" containerID="22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a"
Sep 11 16:48:32.137467 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.137449    2576 scope.go:117] "RemoveContainer" containerID="046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd"
Sep 11 16:48:32.144776 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.144759    2576 scope.go:117] "RemoveContainer" containerID="5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55"
Sep 11 16:48:32.151945 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.151917    2576 scope.go:117] "RemoveContainer" containerID="22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a"
Sep 11 16:48:32.152209 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:32.152188    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a\": container with ID starting with 22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a not found: ID does not exist" containerID="22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a"
Sep 11 16:48:32.152312 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.152221    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a"} err="failed to get container status \"22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a\": rpc error: code = NotFound desc = could not find container \"22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a\": container with ID starting with 22c23e389d6709bcd7bce6b5a2e7b7a2672c2e80fc904abfd985746464312c6a not found: ID does not exist"
Sep 11 16:48:32.152312 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.152244    2576 scope.go:117] "RemoveContainer" containerID="046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd"
Sep 11 16:48:32.152530 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:32.152513    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd\": container with ID starting with 046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd not found: ID does not exist" containerID="046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd"
Sep 11 16:48:32.152567 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.152536    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd"} err="failed to get container status \"046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd\": rpc error: code = NotFound desc = could not find container \"046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd\": container with ID starting with 046a5c4a42d84479c8c56adf62af8f34098ca048d4eff6c31bf87b8c1c532cfd not found: ID does not exist"
Sep 11 16:48:32.152567 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.152551    2576 scope.go:117] "RemoveContainer" containerID="5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55"
Sep 11 16:48:32.152791 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:32.152772    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55\": container with ID starting with 5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55 not found: ID does not exist" containerID="5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55"
Sep 11 16:48:32.152833 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.152797    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55"} err="failed to get container status \"5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55\": rpc error: code = NotFound desc = could not find container \"5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55\": container with ID starting with 5fa5311eeff51d6f5af023287146633678577a7d6abc9f62fd8664bd4262da55 not found: ID does not exist"
Sep 11 16:48:32.164598 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.164575    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"]
Sep 11 16:48:32.174493 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.174474    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc"]
Sep 11 16:48:32.603305 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:32.603273    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" path="/var/lib/kubelet/pods/89dd3d30-20fa-4570-95b2-d4bcec00cb5e/volumes"
Sep 11 16:48:34.866333 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:34.866296    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:48:43.831691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:43.831661    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:43.832093 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:43.832031    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"]
Sep 11 16:48:43.832402 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:43.832367    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" containerID="cri-o://153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f" gracePeriod=30
Sep 11 16:48:43.832531 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:43.832418    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kube-rbac-proxy" containerID="cri-o://c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42" gracePeriod=30
Sep 11 16:48:43.931097 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:43.931066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:44.176417 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:44.176328    2576 generic.go:356] "Generic (PLEG): container finished" podID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerID="c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42" exitCode=2
Sep 11 16:48:44.176417 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:44.176399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" event={"ID":"047b51d0-01e5-40b7-b5b7-2dec5c50aa03","Type":"ContainerDied","Data":"c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42"}
Sep 11 16:48:44.867030 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:44.867003    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:48:46.774314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:46.774278    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 11 16:48:46.778651 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:46.778619    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 11 16:48:48.575438 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.575414    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:48:48.658754 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.658675    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf5zh\" (UniqueName: \"kubernetes.io/projected/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-api-access-wf5zh\") pod \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") "
Sep 11 16:48:48.658754 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.658727    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kserve-provision-location\") pod \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") "
Sep 11 16:48:48.658754 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.658752    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-cabundle-cert\") pod \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") "
Sep 11 16:48:48.659107 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.659074    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "047b51d0-01e5-40b7-b5b7-2dec5c50aa03" (UID: "047b51d0-01e5-40b7-b5b7-2dec5c50aa03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:48:48.659173 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.659156    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "047b51d0-01e5-40b7-b5b7-2dec5c50aa03" (UID: "047b51d0-01e5-40b7-b5b7-2dec5c50aa03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:48:48.660914 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.660895    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-api-access-wf5zh" (OuterVolumeSpecName: "kube-api-access-wf5zh") pod "047b51d0-01e5-40b7-b5b7-2dec5c50aa03" (UID: "047b51d0-01e5-40b7-b5b7-2dec5c50aa03"). InnerVolumeSpecName "kube-api-access-wf5zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:48:48.759759 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.759726    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-proxy-tls\") pod \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") "
Sep 11 16:48:48.759759 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.759764    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-rbac-proxy-sar-config\") pod \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\" (UID: \"047b51d0-01e5-40b7-b5b7-2dec5c50aa03\") "
Sep 11 16:48:48.760033 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.759963    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf5zh\" (UniqueName: \"kubernetes.io/projected/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-api-access-wf5zh\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.760033 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.759981    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.760033 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.759994    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.760208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.760191    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "047b51d0-01e5-40b7-b5b7-2dec5c50aa03" (UID: "047b51d0-01e5-40b7-b5b7-2dec5c50aa03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:48:48.761983 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.761961    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "047b51d0-01e5-40b7-b5b7-2dec5c50aa03" (UID: "047b51d0-01e5-40b7-b5b7-2dec5c50aa03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:48:48.860444 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.860417    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.860444 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.860442    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047b51d0-01e5-40b7-b5b7-2dec5c50aa03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.996242 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:48.996184    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:49.080316 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080280    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"]
Sep 11 16:48:49.080614 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080597    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="storage-initializer"
Sep 11 16:48:49.080677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080620    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="storage-initializer"
Sep 11 16:48:49.080677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080635    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container"
Sep 11 16:48:49.080677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080641    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container"
Sep 11 16:48:49.080677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080653    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kube-rbac-proxy"
Sep 11 16:48:49.080677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080659    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kube-rbac-proxy"
Sep 11 16:48:49.080677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080675    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kube-rbac-proxy"
Sep 11 16:48:49.080677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080680    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kube-rbac-proxy"
Sep 11 16:48:49.080911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080687    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="storage-initializer"
Sep 11 16:48:49.080911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080693    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="storage-initializer"
Sep 11 16:48:49.080911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080704    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container"
Sep 11 16:48:49.080911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080712    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container"
Sep 11 16:48:49.080911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080783    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kube-rbac-proxy"
Sep 11 16:48:49.080911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080793    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kserve-container"
Sep 11 16:48:49.080911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080801    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerName="kserve-container"
Sep 11 16:48:49.080911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.080808    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="89dd3d30-20fa-4570-95b2-d4bcec00cb5e" containerName="kube-rbac-proxy"
Sep 11 16:48:49.084093 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.084077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.087691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.087658    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 11 16:48:49.087691 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.087677    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 11 16:48:49.102782 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.102757    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"]
Sep 11 16:48:49.108256 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.108235    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:49.147833 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.147800    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"]
Sep 11 16:48:49.151041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.151024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.153600 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.153578    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 11 16:48:49.162644 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.162737 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.162737 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.162737 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpd79\" (UniqueName: \"kubernetes.io/projected/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-api-access-cpd79\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.162905 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.162905 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cecc23-ac05-468d-9081-277fb4618ab5-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.162905 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.163091 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162905    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"]
Sep 11 16:48:49.163091 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.163091 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.162985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796gp\" (UniqueName: \"kubernetes.io/projected/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-api-access-796gp\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.163091 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.163020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cecc23-ac05-468d-9081-277fb4618ab5-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.197535 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.197504    2576 generic.go:356] "Generic (PLEG): container finished" podID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" containerID="153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f" exitCode=0
Sep 11 16:48:49.197704 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.197574    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"
Sep 11 16:48:49.197704 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.197585    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" event={"ID":"047b51d0-01e5-40b7-b5b7-2dec5c50aa03","Type":"ContainerDied","Data":"153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f"}
Sep 11 16:48:49.197704 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.197656    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb" event={"ID":"047b51d0-01e5-40b7-b5b7-2dec5c50aa03","Type":"ContainerDied","Data":"6f36a4944b5e4630d43a44a3775bbdc9978a267d7d4b6be1ecff53b2e31b212e"}
Sep 11 16:48:49.197704 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.197691    2576 scope.go:117] "RemoveContainer" containerID="c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42"
Sep 11 16:48:49.206656 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.206638    2576 scope.go:117] "RemoveContainer" containerID="153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f"
Sep 11 16:48:49.214375 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.214355    2576 scope.go:117] "RemoveContainer" containerID="2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869"
Sep 11 16:48:49.221946 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.221916    2576 scope.go:117] "RemoveContainer" containerID="c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42"
Sep 11 16:48:49.222367 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:49.222330    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42\": container with ID starting with c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42 not found: ID does not exist" containerID="c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42"
Sep 11 16:48:49.222489 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.222369    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42"} err="failed to get container status \"c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42\": rpc error: code = NotFound desc = could not find container \"c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42\": container with ID starting with c5cf0d849681a4c8dd3acac6905c737c285e1789163beac6a34ce1240b933a42 not found: ID does not exist"
Sep 11 16:48:49.222489 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.222395    2576 scope.go:117] "RemoveContainer" containerID="153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f"
Sep 11 16:48:49.223274 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:49.222845    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f\": container with ID starting with 153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f not found: ID does not exist" containerID="153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f"
Sep 11 16:48:49.223274 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.222879    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f"} err="failed to get container status \"153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f\": rpc error: code = NotFound desc = could not find container \"153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f\": container with ID starting with 153f1c9710cda5d056397e75631c43c073747632c1a5f46b775d0e2663e7f35f not found: ID does not exist"
Sep 11 16:48:49.223274 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.222902    2576 scope.go:117] "RemoveContainer" containerID="2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869"
Sep 11 16:48:49.223609 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:49.223294    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869\": container with ID starting with 2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869 not found: ID does not exist" containerID="2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869"
Sep 11 16:48:49.223609 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.223335    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869"} err="failed to get container status \"2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869\": rpc error: code = NotFound desc = could not find container \"2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869\": container with ID starting with 2e1e1e8f7bfd0d69eb8c95f48e55a48f4ff8832c6d7d2d61abc334317560a869 not found: ID does not exist"
Sep 11 16:48:49.224687 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.224669    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"]
Sep 11 16:48:49.235975 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.235928    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb"]
Sep 11 16:48:49.263486 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.263417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.263486 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.263453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.263652 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.263486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpd79\" (UniqueName: \"kubernetes.io/projected/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-api-access-cpd79\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.263652 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.263527    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.263652 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.263554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cecc23-ac05-468d-9081-277fb4618ab5-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.263652 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.263584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.263866 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:49.263686    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 11 16:48:49.263866 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:49.263765    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5cecc23-ac05-468d-9081-277fb4618ab5-proxy-tls podName:d5cecc23-ac05-468d-9081-277fb4618ab5 nodeName:}" failed. No retries permitted until 2026-09-11 16:48:49.763741983 +0000 UTC m=+1923.673543138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d5cecc23-ac05-468d-9081-277fb4618ab5-proxy-tls") pod "isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" (UID: "d5cecc23-ac05-468d-9081-277fb4618ab5") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 11 16:48:49.264066 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.263929    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.264066 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.264040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.264246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.264096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796gp\" (UniqueName: \"kubernetes.io/projected/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-api-access-796gp\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.264246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.264131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cecc23-ac05-468d-9081-277fb4618ab5-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.264246 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.264173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.264401 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.264281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.264401 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.264283    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.264401 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.264372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.264690 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.264672    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.264730 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.264676    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cecc23-ac05-468d-9081-277fb4618ab5-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.266239 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.266221    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.276615 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.276594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpd79\" (UniqueName: \"kubernetes.io/projected/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-api-access-cpd79\") pod \"isvc-canary-promote-predictor-7cf78f884f-s5xm7\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.277041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.277020    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:49.277954 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.277920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-796gp\" (UniqueName: \"kubernetes.io/projected/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-api-access-796gp\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.460765 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.460729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:49.515165 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.515087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:49.737682 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.737648    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"]
Sep 11 16:48:49.740980 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:48:49.740949    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d15b9bf_969d_44a0_bec6_ee3304e16fc3.slice/crio-dd55f35e6cde6394b9d358dbf1b9d5cf1144437257156e83f9eca5cc7ed90504 WatchSource:0}: Error finding container dd55f35e6cde6394b9d358dbf1b9d5cf1144437257156e83f9eca5cc7ed90504: Status 404 returned error can't find the container with id dd55f35e6cde6394b9d358dbf1b9d5cf1144437257156e83f9eca5cc7ed90504
Sep 11 16:48:49.769027 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.768972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cecc23-ac05-468d-9081-277fb4618ab5-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.771300 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.771276    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cecc23-ac05-468d-9081-277fb4618ab5-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:49.993339 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:49.993297    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:50.204342 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:50.204299    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" event={"ID":"4d15b9bf-969d-44a0-bec6-ee3304e16fc3","Type":"ContainerStarted","Data":"6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922"}
Sep 11 16:48:50.204490 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:50.204370    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" event={"ID":"4d15b9bf-969d-44a0-bec6-ee3304e16fc3","Type":"ContainerStarted","Data":"dd55f35e6cde6394b9d358dbf1b9d5cf1144437257156e83f9eca5cc7ed90504"}
Sep 11 16:48:50.275039 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:50.275003    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"]
Sep 11 16:48:50.277711 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:48:50.277683    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cecc23_ac05_468d_9081_277fb4618ab5.slice/crio-4630a238e7198d4ae557f7299c5bb78cf007e9d715633fa2be1f5ae869446f42 WatchSource:0}: Error finding container 4630a238e7198d4ae557f7299c5bb78cf007e9d715633fa2be1f5ae869446f42: Status 404 returned error can't find the container with id 4630a238e7198d4ae557f7299c5bb78cf007e9d715633fa2be1f5ae869446f42
Sep 11 16:48:50.603654 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:50.603560    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="047b51d0-01e5-40b7-b5b7-2dec5c50aa03" path="/var/lib/kubelet/pods/047b51d0-01e5-40b7-b5b7-2dec5c50aa03/volumes"
Sep 11 16:48:51.209986 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:51.209922    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" event={"ID":"d5cecc23-ac05-468d-9081-277fb4618ab5","Type":"ContainerStarted","Data":"1a40531c2b09f9c65449d044a07bedab02cd7ecb96fa6f42b2d2fb45f8d867c9"}
Sep 11 16:48:51.210501 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:51.210007    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" event={"ID":"d5cecc23-ac05-468d-9081-277fb4618ab5","Type":"ContainerStarted","Data":"4630a238e7198d4ae557f7299c5bb78cf007e9d715633fa2be1f5ae869446f42"}
Sep 11 16:48:52.941873 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:52.941840    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"]
Sep 11 16:48:52.942437 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:52.942189    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" containerID="cri-o://7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59" gracePeriod=30
Sep 11 16:48:52.942437 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:52.942236    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kube-rbac-proxy" containerID="cri-o://07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431" gracePeriod=30
Sep 11 16:48:52.948138 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:52.948090    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:53.222000 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:53.221971    2576 generic.go:356] "Generic (PLEG): container finished" podID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerID="07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431" exitCode=2
Sep 11 16:48:53.222175 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:53.222016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" event={"ID":"71527f2c-27fe-4c7f-8a6f-aa89c897dc52","Type":"ContainerDied","Data":"07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431"}
Sep 11 16:48:54.227424 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:54.227394    2576 generic.go:356] "Generic (PLEG): container finished" podID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerID="6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922" exitCode=0
Sep 11 16:48:54.227838 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:54.227468    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" event={"ID":"4d15b9bf-969d-44a0-bec6-ee3304e16fc3","Type":"ContainerDied","Data":"6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922"}
Sep 11 16:48:54.229098 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:54.229074    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerID="1a40531c2b09f9c65449d044a07bedab02cd7ecb96fa6f42b2d2fb45f8d867c9" exitCode=0
Sep 11 16:48:54.229192 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:54.229129    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" event={"ID":"d5cecc23-ac05-468d-9081-277fb4618ab5","Type":"ContainerDied","Data":"1a40531c2b09f9c65449d044a07bedab02cd7ecb96fa6f42b2d2fb45f8d867c9"}
Sep 11 16:48:54.861654 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:54.861607    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 11 16:48:54.866000 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:54.865976    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 11 16:48:55.235326 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:55.235294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" event={"ID":"d5cecc23-ac05-468d-9081-277fb4618ab5","Type":"ContainerStarted","Data":"2d6e30bf4904f7fae762d5e538d7365e61ffeb1424eb0becccba14efc0b79673"}
Sep 11 16:48:55.235767 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:55.235366    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" event={"ID":"d5cecc23-ac05-468d-9081-277fb4618ab5","Type":"ContainerStarted","Data":"0b31f850eb3cb18c2b470696bc332377d3930a634c63e0c9f87230525190ec37"}
Sep 11 16:48:55.235767 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:55.235686    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:55.235879 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:55.235858    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:48:55.237298 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:55.237270    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:48:55.237539 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:55.237516    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" event={"ID":"4d15b9bf-969d-44a0-bec6-ee3304e16fc3","Type":"ContainerStarted","Data":"631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae"}
Sep 11 16:48:55.237601 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:55.237564    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" event={"ID":"4d15b9bf-969d-44a0-bec6-ee3304e16fc3","Type":"ContainerStarted","Data":"5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd"}
Sep 11 16:48:55.237735 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:55.237722    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:55.259445 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:55.259400    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podStartSLOduration=2.5964744939999997 podStartE2EDuration="6.259387602s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.662913108s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:55.25791391 +0000 UTC m=+1929.167715071" watchObservedRunningTime="2026-09-11 16:48:55.259387602 +0000 UTC m=+1929.169188765"
Sep 11 16:48:56.241231 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:56.241192    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:48:56.241711 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:56.241417    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:48:56.242614 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:56.242586    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:48:57.244515 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:57.244463    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:48:57.984618 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:57.984593    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:48:58.010233 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.010141    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podStartSLOduration=5.374061362 podStartE2EDuration="9.010120526s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.636059164s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:55.279021951 +0000 UTC m=+1929.188823127" watchObservedRunningTime="2026-09-11 16:48:58.010120526 +0000 UTC m=+1931.919921689"
Sep 11 16:48:58.036915 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.036877    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtw6x\" (UniqueName: \"kubernetes.io/projected/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-api-access-xtw6x\") pod \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") "
Sep 11 16:48:58.037106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.036963    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kserve-provision-location\") pod \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") "
Sep 11 16:48:58.037106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.036991    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-rbac-proxy-sar-config\") pod \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") "
Sep 11 16:48:58.037106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.037029    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-cabundle-cert\") pod \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") "
Sep 11 16:48:58.037106 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.037054    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-proxy-tls\") pod \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\" (UID: \"71527f2c-27fe-4c7f-8a6f-aa89c897dc52\") "
Sep 11 16:48:58.037347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.037311    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71527f2c-27fe-4c7f-8a6f-aa89c897dc52" (UID: "71527f2c-27fe-4c7f-8a6f-aa89c897dc52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:48:58.037483 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.037443    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71527f2c-27fe-4c7f-8a6f-aa89c897dc52" (UID: "71527f2c-27fe-4c7f-8a6f-aa89c897dc52"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:48:58.037606 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.037579    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71527f2c-27fe-4c7f-8a6f-aa89c897dc52" (UID: "71527f2c-27fe-4c7f-8a6f-aa89c897dc52"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:48:58.039760 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.039730    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-api-access-xtw6x" (OuterVolumeSpecName: "kube-api-access-xtw6x") pod "71527f2c-27fe-4c7f-8a6f-aa89c897dc52" (UID: "71527f2c-27fe-4c7f-8a6f-aa89c897dc52"). InnerVolumeSpecName "kube-api-access-xtw6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:48:58.040065 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.040047    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71527f2c-27fe-4c7f-8a6f-aa89c897dc52" (UID: "71527f2c-27fe-4c7f-8a6f-aa89c897dc52"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:48:58.094512 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.094467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:58.136582 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.136545    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"]
Sep 11 16:48:58.137037 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.137017    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kube-rbac-proxy"
Sep 11 16:48:58.137037 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.137036    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kube-rbac-proxy"
Sep 11 16:48:58.137195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.137059    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="storage-initializer"
Sep 11 16:48:58.137195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.137067    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="storage-initializer"
Sep 11 16:48:58.137195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.137076    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container"
Sep 11 16:48:58.137195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.137082    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container"
Sep 11 16:48:58.137195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.137145    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kserve-container"
Sep 11 16:48:58.137195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.137154    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerName="kube-rbac-proxy"
Sep 11 16:48:58.138233 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.138210    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtw6x\" (UniqueName: \"kubernetes.io/projected/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-api-access-xtw6x\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:58.138233 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.138233    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:58.138352 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.138242    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:58.138352 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.138253    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:58.138352 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.138262    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71527f2c-27fe-4c7f-8a6f-aa89c897dc52-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:48:58.140512 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.140496    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.143111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.143085    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 11 16:48:58.143672 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.143655    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 11 16:48:58.151092 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.151071    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"]
Sep 11 16:48:58.239402 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.239370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7btc\" (UniqueName: \"kubernetes.io/projected/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-api-access-l7btc\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.239402 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.239403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.239674 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.239438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.239674 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.239486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.239674 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.239506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.249904 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.249870    2576 generic.go:356] "Generic (PLEG): container finished" podID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" containerID="7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59" exitCode=0
Sep 11 16:48:58.250366 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.249924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" event={"ID":"71527f2c-27fe-4c7f-8a6f-aa89c897dc52","Type":"ContainerDied","Data":"7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59"}
Sep 11 16:48:58.250366 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.249961    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"
Sep 11 16:48:58.250366 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.250005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls" event={"ID":"71527f2c-27fe-4c7f-8a6f-aa89c897dc52","Type":"ContainerDied","Data":"c0192f75e273ced377c16561e79f78f7f67ff2de99e184caee6d65a505883f56"}
Sep 11 16:48:58.250366 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.250032    2576 scope.go:117] "RemoveContainer" containerID="07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431"
Sep 11 16:48:58.259244 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.259228    2576 scope.go:117] "RemoveContainer" containerID="7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59"
Sep 11 16:48:58.266586 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.266571    2576 scope.go:117] "RemoveContainer" containerID="41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075"
Sep 11 16:48:58.269677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.269656    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:58.274172 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.274156    2576 scope.go:117] "RemoveContainer" containerID="07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431"
Sep 11 16:48:58.274433 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:58.274412    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431\": container with ID starting with 07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431 not found: ID does not exist" containerID="07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431"
Sep 11 16:48:58.274483 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.274442    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431"} err="failed to get container status \"07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431\": rpc error: code = NotFound desc = could not find container \"07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431\": container with ID starting with 07d8530b8d4bec46e99fb1ce9ab9c54732801071b46974f79debd92c142d1431 not found: ID does not exist"
Sep 11 16:48:58.274483 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.274459    2576 scope.go:117] "RemoveContainer" containerID="7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59"
Sep 11 16:48:58.274691 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:58.274675    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59\": container with ID starting with 7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59 not found: ID does not exist" containerID="7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59"
Sep 11 16:48:58.274732 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.274707    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59"} err="failed to get container status \"7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59\": rpc error: code = NotFound desc = could not find container \"7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59\": container with ID starting with 7202739fd885c5fb71cd4a20ca328352580ec9c757e1de71cc1e0652d0a83d59 not found: ID does not exist"
Sep 11 16:48:58.274732 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.274722    2576 scope.go:117] "RemoveContainer" containerID="41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075"
Sep 11 16:48:58.274923 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:48:58.274909    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075\": container with ID starting with 41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075 not found: ID does not exist" containerID="41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075"
Sep 11 16:48:58.275092 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.274925    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075"} err="failed to get container status \"41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075\": rpc error: code = NotFound desc = could not find container \"41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075\": container with ID starting with 41b44587153d5c3a9bb96304b3c5c21c8e4caca3672a09b21c26e329866d6075 not found: ID does not exist"
Sep 11 16:48:58.278581 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.278561    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"]
Sep 11 16:48:58.283125 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.283098    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls"]
Sep 11 16:48:58.340673 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.340593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.340673 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.340637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.340882 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.340685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.340882 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.340713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.340882 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.340767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7btc\" (UniqueName: \"kubernetes.io/projected/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-api-access-l7btc\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.341138 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.341116    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.341353 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.341333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.341419 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.341400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.343279 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.343262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.350394 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.350375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7btc\" (UniqueName: \"kubernetes.io/projected/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-api-access-l7btc\") pod \"isvc-tensorflow-predictor-547dfc5669-lf8vp\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.452047 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.452015    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:48:58.602724 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.602629    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71527f2c-27fe-4c7f-8a6f-aa89c897dc52" path="/var/lib/kubelet/pods/71527f2c-27fe-4c7f-8a6f-aa89c897dc52/volumes"
Sep 11 16:48:58.728540 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:58.728429    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"]
Sep 11 16:48:58.730570 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:48:58.730545    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6454b6_96bb_4ec5_8e04_d95e84ae78ce.slice/crio-6f493fff0817480ab519f2185293813ecbe2afee902ec4a9695f68ea848992cf WatchSource:0}: Error finding container 6f493fff0817480ab519f2185293813ecbe2afee902ec4a9695f68ea848992cf: Status 404 returned error can't find the container with id 6f493fff0817480ab519f2185293813ecbe2afee902ec4a9695f68ea848992cf
Sep 11 16:48:59.256343 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:59.256305    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" event={"ID":"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce","Type":"ContainerStarted","Data":"cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e"}
Sep 11 16:48:59.256806 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:48:59.256378    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" event={"ID":"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce","Type":"ContainerStarted","Data":"6f493fff0817480ab519f2185293813ecbe2afee902ec4a9695f68ea848992cf"}
Sep 11 16:49:01.245904 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:01.245870    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:49:01.246410 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:01.246385    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:49:02.248761 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:02.248734    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:49:02.249310 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:02.249284    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:49:04.278895 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:04.278859    2576 generic.go:356] "Generic (PLEG): container finished" podID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerID="cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e" exitCode=0
Sep 11 16:49:04.279313 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:04.278953    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" event={"ID":"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce","Type":"ContainerDied","Data":"cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e"}
Sep 11 16:49:04.280210 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:04.280194    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:49:08.308464 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:08.308370    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" event={"ID":"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce","Type":"ContainerStarted","Data":"c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4"}
Sep 11 16:49:08.308464 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:08.308460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" event={"ID":"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce","Type":"ContainerStarted","Data":"52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c"}
Sep 11 16:49:08.308883 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:08.308761    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:49:08.333148 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:08.333094    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podStartSLOduration=1.911208486 podStartE2EDuration="10.333080891s" totalImagesPullingTime="3.75554308s" totalInitContainerRuntime="4.666329325s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:49:08.331874386 +0000 UTC m=+1942.241675547" watchObservedRunningTime="2026-09-11 16:49:08.333080891 +0000 UTC m=+1942.242882083"
Sep 11 16:49:09.312533 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:09.312491    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:49:09.313709 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:09.313682    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:49:10.316197 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:10.316155    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:49:11.246872 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:11.246836    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:49:12.249550 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:12.249515    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:49:15.320918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:15.320884    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:49:15.321567 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:15.321538    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 11 16:49:21.247360 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:21.247320    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:49:22.249529 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:22.249487    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:49:25.322331 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:25.322300    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:49:31.246360 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:31.246318    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:49:32.249622 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:32.249582    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:49:39.650891 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:39.650794    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"]
Sep 11 16:49:39.651333 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:39.651093    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kserve-container" containerID="cri-o://52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c" gracePeriod=30
Sep 11 16:49:39.651333 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:39.651138    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy" containerID="cri-o://c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4" gracePeriod=30
Sep 11 16:49:39.656919 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:39.656894    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:49:40.316592 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:40.316552    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 11 16:49:40.440780 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:40.440747    2576 generic.go:356] "Generic (PLEG): container finished" podID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerID="c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4" exitCode=2
Sep 11 16:49:40.440976 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:40.440817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" event={"ID":"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce","Type":"ContainerDied","Data":"c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4"}
Sep 11 16:49:41.246731 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:41.246695    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:49:42.249911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:42.249875    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:49:44.763866 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:44.763832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:49:45.049714 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:45.049637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:49:45.316531 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:45.316433    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 11 16:49:50.317217 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:50.317173    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 11 16:49:50.317612 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:50.317293    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:49:51.247149 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:51.247109    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:49:52.249182 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:52.249145    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:49:55.316392 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:49:55.316347    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 11 16:50:00.316788 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:00.316739    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 11 16:50:01.246369 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:01.246331    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:50:02.250003 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:02.249962    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:50:05.316786 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:05.316743    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 11 16:50:10.301498 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.301469    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:50:10.322693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.322658    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-cabundle-cert\") pod \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") "
Sep 11 16:50:10.322693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.322704    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kserve-provision-location\") pod \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") "
Sep 11 16:50:10.322911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.322737    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7btc\" (UniqueName: \"kubernetes.io/projected/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-api-access-l7btc\") pod \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") "
Sep 11 16:50:10.322911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.322787    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-rbac-proxy-sar-config\") pod \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") "
Sep 11 16:50:10.322911 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.322890    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-proxy-tls\") pod \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\" (UID: \"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce\") "
Sep 11 16:50:10.323184 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.323158    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" (UID: "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:50:10.323258 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.323234    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" (UID: "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:50:10.325206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.325180    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" (UID: "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:50:10.325206 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.325194    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-api-access-l7btc" (OuterVolumeSpecName: "kube-api-access-l7btc") pod "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" (UID: "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce"). InnerVolumeSpecName "kube-api-access-l7btc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:50:10.334352 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.334323    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" (UID: "4e6454b6-96bb-4ec5-8e04-d95e84ae78ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:10.424250 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.424151    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:10.424250 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.424185    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:10.424250 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.424195    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:10.424250 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.424204    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7btc\" (UniqueName: \"kubernetes.io/projected/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-api-access-l7btc\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:10.424250 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.424213    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:10.562267 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.562235    2576 generic.go:356] "Generic (PLEG): container finished" podID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerID="52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c" exitCode=137
Sep 11 16:50:10.562452 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.562297    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"
Sep 11 16:50:10.562452 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.562327    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" event={"ID":"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce","Type":"ContainerDied","Data":"52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c"}
Sep 11 16:50:10.562452 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.562412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp" event={"ID":"4e6454b6-96bb-4ec5-8e04-d95e84ae78ce","Type":"ContainerDied","Data":"6f493fff0817480ab519f2185293813ecbe2afee902ec4a9695f68ea848992cf"}
Sep 11 16:50:10.562452 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.562441    2576 scope.go:117] "RemoveContainer" containerID="c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4"
Sep 11 16:50:10.570888 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.570869    2576 scope.go:117] "RemoveContainer" containerID="52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c"
Sep 11 16:50:10.578511 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.578490    2576 scope.go:117] "RemoveContainer" containerID="cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e"
Sep 11 16:50:10.585804 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.585784    2576 scope.go:117] "RemoveContainer" containerID="c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4"
Sep 11 16:50:10.586139 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:50:10.586121    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4\": container with ID starting with c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4 not found: ID does not exist" containerID="c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4"
Sep 11 16:50:10.586199 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.586151    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4"} err="failed to get container status \"c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4\": rpc error: code = NotFound desc = could not find container \"c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4\": container with ID starting with c2c5d2a24f83ec9ae6dfeeeed1bf74edb94cc9e144aa21a86ed67ee6ecd55ef4 not found: ID does not exist"
Sep 11 16:50:10.586199 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.586172    2576 scope.go:117] "RemoveContainer" containerID="52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c"
Sep 11 16:50:10.586404 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:50:10.586388    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c\": container with ID starting with 52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c not found: ID does not exist" containerID="52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c"
Sep 11 16:50:10.586446 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.586408    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c"} err="failed to get container status \"52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c\": rpc error: code = NotFound desc = could not find container \"52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c\": container with ID starting with 52b29e01ac1b21b6b9f175e6ecb459273a4f7ba7c06b49d90ab5d3d68b2c6a6c not found: ID does not exist"
Sep 11 16:50:10.586446 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.586422    2576 scope.go:117] "RemoveContainer" containerID="cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e"
Sep 11 16:50:10.586649 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:50:10.586632    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e\": container with ID starting with cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e not found: ID does not exist" containerID="cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e"
Sep 11 16:50:10.586693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.586652    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e"} err="failed to get container status \"cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e\": rpc error: code = NotFound desc = could not find container \"cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e\": container with ID starting with cfe064e136dea96c13be393b6466c0a7f300114b1179ed8bfd4ce080fb8a5c3e not found: ID does not exist"
Sep 11 16:50:10.595468 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.595431    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"]
Sep 11 16:50:10.603888 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:10.603863    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp"]
Sep 11 16:50:11.247062 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:11.247024    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:50:12.250093 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:12.250059    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:50:12.603367 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:12.603285    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" path="/var/lib/kubelet/pods/4e6454b6-96bb-4ec5-8e04-d95e84ae78ce/volumes"
Sep 11 16:50:21.247101 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:21.247071    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:50:21.407599 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:21.407560    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"]
Sep 11 16:50:21.407843 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:21.407817    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" containerID="cri-o://5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd" gracePeriod=30
Sep 11 16:50:21.407964 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:21.407863    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kube-rbac-proxy" containerID="cri-o://631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae" gracePeriod=30
Sep 11 16:50:21.612566 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:21.612527    2576 generic.go:356] "Generic (PLEG): container finished" podID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerID="631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae" exitCode=2
Sep 11 16:50:21.612566 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:21.612561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" event={"ID":"4d15b9bf-969d-44a0-bec6-ee3304e16fc3","Type":"ContainerDied","Data":"631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae"}
Sep 11 16:50:22.244700 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:22.244659    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 11 16:50:22.250142 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:22.250117    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 11 16:50:24.140115 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:24.140076    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:24.778247 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:24.778210    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:26.150449 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.150424    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:50:26.247677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.247596    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-cabundle-cert\") pod \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") "
Sep 11 16:50:26.247677 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.247648    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-rbac-proxy-sar-config\") pod \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") "
Sep 11 16:50:26.247889 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.247697    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-proxy-tls\") pod \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") "
Sep 11 16:50:26.247889 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.247730    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kserve-provision-location\") pod \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") "
Sep 11 16:50:26.247889 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.247754    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpd79\" (UniqueName: \"kubernetes.io/projected/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-api-access-cpd79\") pod \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\" (UID: \"4d15b9bf-969d-44a0-bec6-ee3304e16fc3\") "
Sep 11 16:50:26.248119 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.248012    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d15b9bf-969d-44a0-bec6-ee3304e16fc3" (UID: "4d15b9bf-969d-44a0-bec6-ee3304e16fc3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:50:26.248119 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.248026    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d15b9bf-969d-44a0-bec6-ee3304e16fc3" (UID: "4d15b9bf-969d-44a0-bec6-ee3304e16fc3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:50:26.248119 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.248100    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d15b9bf-969d-44a0-bec6-ee3304e16fc3" (UID: "4d15b9bf-969d-44a0-bec6-ee3304e16fc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:26.250044 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.250013    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d15b9bf-969d-44a0-bec6-ee3304e16fc3" (UID: "4d15b9bf-969d-44a0-bec6-ee3304e16fc3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:50:26.250202 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.250183    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-api-access-cpd79" (OuterVolumeSpecName: "kube-api-access-cpd79") pod "4d15b9bf-969d-44a0-bec6-ee3304e16fc3" (UID: "4d15b9bf-969d-44a0-bec6-ee3304e16fc3"). InnerVolumeSpecName "kube-api-access-cpd79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:50:26.302787 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.302754    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:26.349203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.349175    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.349203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.349199    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpd79\" (UniqueName: \"kubernetes.io/projected/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-api-access-cpd79\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.349203 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.349208    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.349434 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.349219    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.349434 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.349228    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d15b9bf-969d-44a0-bec6-ee3304e16fc3-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.632226 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.632197    2576 generic.go:356] "Generic (PLEG): container finished" podID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerID="5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd" exitCode=0
Sep 11 16:50:26.632408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.632262    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"
Sep 11 16:50:26.632408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.632260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" event={"ID":"4d15b9bf-969d-44a0-bec6-ee3304e16fc3","Type":"ContainerDied","Data":"5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd"}
Sep 11 16:50:26.632408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.632391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7" event={"ID":"4d15b9bf-969d-44a0-bec6-ee3304e16fc3","Type":"ContainerDied","Data":"dd55f35e6cde6394b9d358dbf1b9d5cf1144437257156e83f9eca5cc7ed90504"}
Sep 11 16:50:26.632564 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.632419    2576 scope.go:117] "RemoveContainer" containerID="631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae"
Sep 11 16:50:26.640802 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.640784    2576 scope.go:117] "RemoveContainer" containerID="5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd"
Sep 11 16:50:26.648262 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.648246    2576 scope.go:117] "RemoveContainer" containerID="6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922"
Sep 11 16:50:26.657455 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.657423    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"]
Sep 11 16:50:26.661103 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.661074    2576 scope.go:117] "RemoveContainer" containerID="631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae"
Sep 11 16:50:26.661659 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:50:26.661638    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae\": container with ID starting with 631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae not found: ID does not exist" containerID="631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae"
Sep 11 16:50:26.661737 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.661668    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae"} err="failed to get container status \"631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae\": rpc error: code = NotFound desc = could not find container \"631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae\": container with ID starting with 631f21cabbf437eb710216b268f21e3fa122a67f25c5268ca8cfb21085deb8ae not found: ID does not exist"
Sep 11 16:50:26.661737 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.661688    2576 scope.go:117] "RemoveContainer" containerID="5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd"
Sep 11 16:50:26.661827 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.661809    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7"]
Sep 11 16:50:26.661980 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:50:26.661931    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd\": container with ID starting with 5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd not found: ID does not exist" containerID="5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd"
Sep 11 16:50:26.662025 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.661990    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd"} err="failed to get container status \"5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd\": rpc error: code = NotFound desc = could not find container \"5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd\": container with ID starting with 5edbe4e98ad393650706e36a8c3ff1239a1894d4cf92cad2e08c5af4e70890cd not found: ID does not exist"
Sep 11 16:50:26.662025 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.662011    2576 scope.go:117] "RemoveContainer" containerID="6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922"
Sep 11 16:50:26.662228 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:50:26.662214    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922\": container with ID starting with 6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922 not found: ID does not exist" containerID="6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922"
Sep 11 16:50:26.662271 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:26.662231    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922"} err="failed to get container status \"6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922\": rpc error: code = NotFound desc = could not find container \"6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922\": container with ID starting with 6a08e8214c85f1be4cb4a6a276f2ede520acf7d1a7f31fd25cbeefa0f014d922 not found: ID does not exist"
Sep 11 16:50:28.602882 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:28.602851    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" path="/var/lib/kubelet/pods/4d15b9bf-969d-44a0-bec6-ee3304e16fc3/volumes"
Sep 11 16:50:31.453802 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.453773    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:31.509616 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509590    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"]
Sep 11 16:50:31.509878 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509866    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kube-rbac-proxy"
Sep 11 16:50:31.509952 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509880    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kube-rbac-proxy"
Sep 11 16:50:31.509952 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509889    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container"
Sep 11 16:50:31.509952 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509894    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container"
Sep 11 16:50:31.509952 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509909    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kserve-container"
Sep 11 16:50:31.509952 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509915    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kserve-container"
Sep 11 16:50:31.509952 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509923    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="storage-initializer"
Sep 11 16:50:31.509952 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509928    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="storage-initializer"
Sep 11 16:50:31.509952 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509953    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy"
Sep 11 16:50:31.510208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509961    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy"
Sep 11 16:50:31.510208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509970    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="storage-initializer"
Sep 11 16:50:31.510208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.509975    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="storage-initializer"
Sep 11 16:50:31.510208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.510025    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kserve-container"
Sep 11 16:50:31.510208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.510036    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kube-rbac-proxy"
Sep 11 16:50:31.510208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.510042    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d15b9bf-969d-44a0-bec6-ee3304e16fc3" containerName="kube-rbac-proxy"
Sep 11 16:50:31.510208 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.510049    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6454b6-96bb-4ec5-8e04-d95e84ae78ce" containerName="kserve-container"
Sep 11 16:50:31.514416 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.514398    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.522394 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.522374    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 11 16:50:31.522751 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.522734    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 11 16:50:31.530072 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.530047    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"]
Sep 11 16:50:31.537139 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.537118    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:31.590252 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.590214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.590436 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.590265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf537059-3c80-40f3-91b2-467db47491bc-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.590436 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.590340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vzrc\" (UniqueName: \"kubernetes.io/projected/bf537059-3c80-40f3-91b2-467db47491bc-kube-api-access-4vzrc\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.590541 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.590436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.590541 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.590477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf537059-3c80-40f3-91b2-467db47491bc-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.691504 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.691463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf537059-3c80-40f3-91b2-467db47491bc-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.691504 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.691509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.691775 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.691637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf537059-3c80-40f3-91b2-467db47491bc-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.691775 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.691697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vzrc\" (UniqueName: \"kubernetes.io/projected/bf537059-3c80-40f3-91b2-467db47491bc-kube-api-access-4vzrc\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.691775 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.691760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.692113 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.692077    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf537059-3c80-40f3-91b2-467db47491bc-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.692215 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.692179    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.692384 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.692364    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.694321 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.694300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf537059-3c80-40f3-91b2-467db47491bc-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.702425 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.702403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vzrc\" (UniqueName: \"kubernetes.io/projected/bf537059-3c80-40f3-91b2-467db47491bc-kube-api-access-4vzrc\") pod \"isvc-triton-predictor-5d7db58676-k4mg9\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:31.823845 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:31.823756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:50:32.108433 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:32.108375    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"]
Sep 11 16:50:32.110665 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:50:32.110638    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf537059_3c80_40f3_91b2_467db47491bc.slice/crio-6aaef21db7460e2d3340aac35a8cff80f1dabfbaf3141516cda29e5482e42492 WatchSource:0}: Error finding container 6aaef21db7460e2d3340aac35a8cff80f1dabfbaf3141516cda29e5482e42492: Status 404 returned error can't find the container with id 6aaef21db7460e2d3340aac35a8cff80f1dabfbaf3141516cda29e5482e42492
Sep 11 16:50:32.656954 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:32.656901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" event={"ID":"bf537059-3c80-40f3-91b2-467db47491bc","Type":"ContainerStarted","Data":"86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a"}
Sep 11 16:50:32.657373 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:32.656984    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" event={"ID":"bf537059-3c80-40f3-91b2-467db47491bc","Type":"ContainerStarted","Data":"6aaef21db7460e2d3340aac35a8cff80f1dabfbaf3141516cda29e5482e42492"}
Sep 11 16:50:36.674382 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:36.674348    2576 generic.go:356] "Generic (PLEG): container finished" podID="bf537059-3c80-40f3-91b2-467db47491bc" containerID="86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a" exitCode=0
Sep 11 16:50:36.674780 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:50:36.674427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" event={"ID":"bf537059-3c80-40f3-91b2-467db47491bc","Type":"ContainerDied","Data":"86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a"}
Sep 11 16:51:54.382056 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:54.382008    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"]
Sep 11 16:51:54.382799 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:54.382329    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" containerID="cri-o://0b31f850eb3cb18c2b470696bc332377d3930a634c63e0c9f87230525190ec37" gracePeriod=30
Sep 11 16:51:54.382799 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:54.382401    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kube-rbac-proxy" containerID="cri-o://2d6e30bf4904f7fae762d5e538d7365e61ffeb1424eb0becccba14efc0b79673" gracePeriod=30
Sep 11 16:51:54.435442 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:54.435404    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:56.092295 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:56.092248    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerID="2d6e30bf4904f7fae762d5e538d7365e61ffeb1424eb0becccba14efc0b79673" exitCode=2
Sep 11 16:51:56.092791 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:56.092321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" event={"ID":"d5cecc23-ac05-468d-9081-277fb4618ab5","Type":"ContainerDied","Data":"2d6e30bf4904f7fae762d5e538d7365e61ffeb1424eb0becccba14efc0b79673"}
Sep 11 16:51:56.241882 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:56.241837    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 11 16:51:59.565281 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.565239    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:59.632854 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.632816    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"]
Sep 11 16:51:59.658439 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.657681    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"]
Sep 11 16:51:59.658439 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.657818    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.661103 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.661078    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 11 16:51:59.661373 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.661271    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:59.661644 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.661620    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 11 16:51:59.697309 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.697283    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"]
Sep 11 16:51:59.713211 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.713186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"]
Sep 11 16:51:59.713211 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.713202    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.716732 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.716708    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 11 16:51:59.752766 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.752739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.752964 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.752790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.752964 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.752822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.752964 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.752860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlfjd\" (UniqueName: \"kubernetes.io/projected/f4634433-246b-452d-a6bc-84ef26b98359-kube-api-access-hlfjd\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.752964 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.752883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7bf\" (UniqueName: \"kubernetes.io/projected/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-api-access-qp7bf\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.752964 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.752911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.752964 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.752955    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.753220 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.753019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4634433-246b-452d-a6bc-84ef26b98359-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.753220 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.753055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4634433-246b-452d-a6bc-84ef26b98359-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.753220 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.753082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.805196 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.805146    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:59.853989 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.853901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.853989 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.853953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.853989 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.853986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4634433-246b-452d-a6bc-84ef26b98359-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.854256 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4634433-246b-452d-a6bc-84ef26b98359-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.854256 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.854256 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.854256 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.854256 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.854256 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlfjd\" (UniqueName: \"kubernetes.io/projected/f4634433-246b-452d-a6bc-84ef26b98359-kube-api-access-hlfjd\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.854256 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:51:59.854215    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 11 16:51:59.854256 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7bf\" (UniqueName: \"kubernetes.io/projected/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-api-access-qp7bf\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.854662 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:51:59.854295    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4634433-246b-452d-a6bc-84ef26b98359-proxy-tls podName:f4634433-246b-452d-a6bc-84ef26b98359 nodeName:}" failed. No retries permitted until 2026-09-11 16:52:00.354273172 +0000 UTC m=+2114.264074315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4634433-246b-452d-a6bc-84ef26b98359-proxy-tls") pod "isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" (UID: "f4634433-246b-452d-a6bc-84ef26b98359") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 11 16:51:59.855134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854727    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.855134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854729    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.855134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.854785    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.855134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.855032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.855347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.855165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.855399 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.855344    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4634433-246b-452d-a6bc-84ef26b98359-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.857019 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.856999    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.868621 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.868600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlfjd\" (UniqueName: \"kubernetes.io/projected/f4634433-246b-452d-a6bc-84ef26b98359-kube-api-access-hlfjd\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:51:59.868758 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.868741    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7bf\" (UniqueName: \"kubernetes.io/projected/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-api-access-qp7bf\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:51:59.914414 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.914383    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:59.997041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:51:59.997008    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:52:00.358985 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:00.358921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4634433-246b-452d-a6bc-84ef26b98359-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:52:00.362045 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:00.362014    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4634433-246b-452d-a6bc-84ef26b98359-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:52:00.626854 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:00.626752    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:52:01.242578 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:01.242523    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 11 16:52:01.246833 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:01.246797    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 11 16:52:02.121236 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.121195    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerID="0b31f850eb3cb18c2b470696bc332377d3930a634c63e0c9f87230525190ec37" exitCode=0
Sep 11 16:52:02.121748 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.121255    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" event={"ID":"d5cecc23-ac05-468d-9081-277fb4618ab5","Type":"ContainerDied","Data":"0b31f850eb3cb18c2b470696bc332377d3930a634c63e0c9f87230525190ec37"}
Sep 11 16:52:02.647478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.647443    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:52:02.677514 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.677477    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-rbac-proxy-sar-config\") pod \"d5cecc23-ac05-468d-9081-277fb4618ab5\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") "
Sep 11 16:52:02.677690 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.677526    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-cabundle-cert\") pod \"d5cecc23-ac05-468d-9081-277fb4618ab5\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") "
Sep 11 16:52:02.677690 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.677564    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-796gp\" (UniqueName: \"kubernetes.io/projected/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-api-access-796gp\") pod \"d5cecc23-ac05-468d-9081-277fb4618ab5\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") "
Sep 11 16:52:02.677690 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.677597    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cecc23-ac05-468d-9081-277fb4618ab5-proxy-tls\") pod \"d5cecc23-ac05-468d-9081-277fb4618ab5\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") "
Sep 11 16:52:02.677690 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.677642    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cecc23-ac05-468d-9081-277fb4618ab5-kserve-provision-location\") pod \"d5cecc23-ac05-468d-9081-277fb4618ab5\" (UID: \"d5cecc23-ac05-468d-9081-277fb4618ab5\") "
Sep 11 16:52:02.677900 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.677875    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5cecc23-ac05-468d-9081-277fb4618ab5" (UID: "d5cecc23-ac05-468d-9081-277fb4618ab5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:52:02.678195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.678160    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cecc23-ac05-468d-9081-277fb4618ab5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5cecc23-ac05-468d-9081-277fb4618ab5" (UID: "d5cecc23-ac05-468d-9081-277fb4618ab5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:02.678195 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.678164    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5cecc23-ac05-468d-9081-277fb4618ab5" (UID: "d5cecc23-ac05-468d-9081-277fb4618ab5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:52:02.682272 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.682245    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5cecc23-ac05-468d-9081-277fb4618ab5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5cecc23-ac05-468d-9081-277fb4618ab5" (UID: "d5cecc23-ac05-468d-9081-277fb4618ab5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:52:02.682418 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.682349    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-api-access-796gp" (OuterVolumeSpecName: "kube-api-access-796gp") pod "d5cecc23-ac05-468d-9081-277fb4618ab5" (UID: "d5cecc23-ac05-468d-9081-277fb4618ab5"). InnerVolumeSpecName "kube-api-access-796gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:52:02.778840 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.778810    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:02.778840 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.778834    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-796gp\" (UniqueName: \"kubernetes.io/projected/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-api-access-796gp\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:02.778840 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.778845    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5cecc23-ac05-468d-9081-277fb4618ab5-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:02.779161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.778854    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5cecc23-ac05-468d-9081-277fb4618ab5-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:02.779161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.778866    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5cecc23-ac05-468d-9081-277fb4618ab5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:02.885634 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.885590    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"]
Sep 11 16:52:02.977316 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:02.977281    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"]
Sep 11 16:52:03.128087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:03.128042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h" event={"ID":"d5cecc23-ac05-468d-9081-277fb4618ab5","Type":"ContainerDied","Data":"4630a238e7198d4ae557f7299c5bb78cf007e9d715633fa2be1f5ae869446f42"}
Sep 11 16:52:03.128561 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:03.128132    2576 scope.go:117] "RemoveContainer" containerID="2d6e30bf4904f7fae762d5e538d7365e61ffeb1424eb0becccba14efc0b79673"
Sep 11 16:52:03.128561 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:03.128138    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"
Sep 11 16:52:03.155621 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:03.155595    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"]
Sep 11 16:52:03.164087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:03.164062    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h"]
Sep 11 16:52:04.604136 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:04.604095    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" path="/var/lib/kubelet/pods/d5cecc23-ac05-468d-9081-277fb4618ab5/volumes"
Sep 11 16:52:29.971220 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:29.970860    2576 scope.go:117] "RemoveContainer" containerID="0b31f850eb3cb18c2b470696bc332377d3930a634c63e0c9f87230525190ec37"
Sep 11 16:52:29.972738 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:52:29.972709    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ff62d5_bd3b_4727_b6c3_cae2cf88d4b0.slice/crio-72385d5fd559a0d90f5d9622c8342de07badf9555ae1d35044012d6815839868 WatchSource:0}: Error finding container 72385d5fd559a0d90f5d9622c8342de07badf9555ae1d35044012d6815839868: Status 404 returned error can't find the container with id 72385d5fd559a0d90f5d9622c8342de07badf9555ae1d35044012d6815839868
Sep 11 16:52:29.973237 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:52:29.973201    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4634433_246b_452d_a6bc_84ef26b98359.slice/crio-7d25a9b0a0b76b575f5b3c4455288124d35c740d74da9a198d0d4f92e7356492 WatchSource:0}: Error finding container 7d25a9b0a0b76b575f5b3c4455288124d35c740d74da9a198d0d4f92e7356492: Status 404 returned error can't find the container with id 7d25a9b0a0b76b575f5b3c4455288124d35c740d74da9a198d0d4f92e7356492
Sep 11 16:52:30.164835 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:30.164791    2576 scope.go:117] "RemoveContainer" containerID="1a40531c2b09f9c65449d044a07bedab02cd7ecb96fa6f42b2d2fb45f8d867c9"
Sep 11 16:52:30.180040 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:30.180012    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:52:30.180153 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:30.180025    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:52:30.190504 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:30.190483    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:52:30.190504 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:30.190492    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:52:30.246432 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:30.246406    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" event={"ID":"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0","Type":"ContainerStarted","Data":"72385d5fd559a0d90f5d9622c8342de07badf9555ae1d35044012d6815839868"}
Sep 11 16:52:30.248372 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:30.248350    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" event={"ID":"f4634433-246b-452d-a6bc-84ef26b98359","Type":"ContainerStarted","Data":"7d25a9b0a0b76b575f5b3c4455288124d35c740d74da9a198d0d4f92e7356492"}
Sep 11 16:52:31.254063 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:31.254022    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" event={"ID":"f4634433-246b-452d-a6bc-84ef26b98359","Type":"ContainerStarted","Data":"4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248"}
Sep 11 16:52:31.255879 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:31.255845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" event={"ID":"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0","Type":"ContainerStarted","Data":"05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f"}
Sep 11 16:52:31.258380 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:31.258346    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" event={"ID":"bf537059-3c80-40f3-91b2-467db47491bc","Type":"ContainerStarted","Data":"a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3"}
Sep 11 16:52:31.258535 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:31.258400    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" event={"ID":"bf537059-3c80-40f3-91b2-467db47491bc","Type":"ContainerStarted","Data":"5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d"}
Sep 11 16:52:31.258535 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:31.258476    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:52:31.322401 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:31.322353    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" podStartSLOduration=2.048013721 podStartE2EDuration="2m0.322332975s" totalImagesPullingTime="1m54.253804583s" totalInitContainerRuntime="4.020514671s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:52:31.321354846 +0000 UTC m=+2145.231156029" watchObservedRunningTime="2026-09-11 16:52:31.322332975 +0000 UTC m=+2145.232134138"
Sep 11 16:52:32.263054 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:32.263026    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:52:32.264247 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:32.264221    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 11 16:52:33.266260 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:33.266216    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 11 16:52:35.274984 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:35.274921    2576 generic.go:356] "Generic (PLEG): container finished" podID="f4634433-246b-452d-a6bc-84ef26b98359" containerID="4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248" exitCode=0
Sep 11 16:52:35.275431 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:35.274993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" event={"ID":"f4634433-246b-452d-a6bc-84ef26b98359","Type":"ContainerDied","Data":"4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248"}
Sep 11 16:52:35.276649 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:35.276628    2576 generic.go:356] "Generic (PLEG): container finished" podID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerID="05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f" exitCode=0
Sep 11 16:52:35.276729 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:35.276678    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" event={"ID":"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0","Type":"ContainerDied","Data":"05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f"}
Sep 11 16:52:36.283873 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:36.283828    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" event={"ID":"f4634433-246b-452d-a6bc-84ef26b98359","Type":"ContainerStarted","Data":"69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912"}
Sep 11 16:52:36.284328 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:36.283901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" event={"ID":"f4634433-246b-452d-a6bc-84ef26b98359","Type":"ContainerStarted","Data":"8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254"}
Sep 11 16:52:36.284328 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:36.284139    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:52:36.286077 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:36.286056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" event={"ID":"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0","Type":"ContainerStarted","Data":"a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59"}
Sep 11 16:52:36.286183 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:36.286102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" event={"ID":"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0","Type":"ContainerStarted","Data":"5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a"}
Sep 11 16:52:36.286290 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:36.286274    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:52:36.311947 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:36.311884    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podStartSLOduration=32.903495925 podStartE2EDuration="37.311870097s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.408374172s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:52:36.310210041 +0000 UTC m=+2150.220011229" watchObservedRunningTime="2026-09-11 16:52:36.311870097 +0000 UTC m=+2150.221671258"
Sep 11 16:52:36.334916 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:36.334869    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podStartSLOduration=33.093266485 podStartE2EDuration="37.334852127s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.241585642s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:52:36.334206641 +0000 UTC m=+2150.244007804" watchObservedRunningTime="2026-09-11 16:52:36.334852127 +0000 UTC m=+2150.244653288"
Sep 11 16:52:37.290741 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:37.290711    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:52:37.291364 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:37.291343    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:52:37.291599 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:37.291574    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 11 16:52:37.291833 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:37.291807    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:52:37.778274 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:37.778238    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:38.270466 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:38.270436    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:52:38.271062 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:38.271042    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:52:38.293817 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:38.293784    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:52:38.294698 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:38.293812    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 11 16:52:43.299490 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:43.299462    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:52:43.299865 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:43.299550    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:52:43.300059 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:43.300035    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:52:43.300174 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:43.300146    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 11 16:52:45.355626 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:45.355588    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"]
Sep 11 16:52:45.356185 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:45.355872    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kserve-container" containerID="cri-o://5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d" gracePeriod=30
Sep 11 16:52:45.356185 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:45.355900    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kube-rbac-proxy" containerID="cri-o://a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3" gracePeriod=30
Sep 11 16:52:45.384128 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:45.384100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:46.328572 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:46.328542    2576 generic.go:356] "Generic (PLEG): container finished" podID="bf537059-3c80-40f3-91b2-467db47491bc" containerID="a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3" exitCode=2
Sep 11 16:52:46.328735 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:46.328618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" event={"ID":"bf537059-3c80-40f3-91b2-467db47491bc","Type":"ContainerDied","Data":"a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3"}
Sep 11 16:52:48.092130 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.092102    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:52:48.270988 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.270954    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vzrc\" (UniqueName: \"kubernetes.io/projected/bf537059-3c80-40f3-91b2-467db47491bc-kube-api-access-4vzrc\") pod \"bf537059-3c80-40f3-91b2-467db47491bc\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") "
Sep 11 16:52:48.270988 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.270987    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-cabundle-cert\") pod \"bf537059-3c80-40f3-91b2-467db47491bc\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") "
Sep 11 16:52:48.271166 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.271018    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-kube-rbac-proxy-sar-config\") pod \"bf537059-3c80-40f3-91b2-467db47491bc\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") "
Sep 11 16:52:48.271166 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.271070    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf537059-3c80-40f3-91b2-467db47491bc-proxy-tls\") pod \"bf537059-3c80-40f3-91b2-467db47491bc\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") "
Sep 11 16:52:48.271166 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.271091    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf537059-3c80-40f3-91b2-467db47491bc-kserve-provision-location\") pod \"bf537059-3c80-40f3-91b2-467db47491bc\" (UID: \"bf537059-3c80-40f3-91b2-467db47491bc\") "
Sep 11 16:52:48.271373 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.271345    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf537059-3c80-40f3-91b2-467db47491bc" (UID: "bf537059-3c80-40f3-91b2-467db47491bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:52:48.271457 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.271367    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf537059-3c80-40f3-91b2-467db47491bc" (UID: "bf537059-3c80-40f3-91b2-467db47491bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:52:48.271612 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.271586    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf537059-3c80-40f3-91b2-467db47491bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf537059-3c80-40f3-91b2-467db47491bc" (UID: "bf537059-3c80-40f3-91b2-467db47491bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:48.273209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.273184    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf537059-3c80-40f3-91b2-467db47491bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf537059-3c80-40f3-91b2-467db47491bc" (UID: "bf537059-3c80-40f3-91b2-467db47491bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:52:48.273270 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.273249    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf537059-3c80-40f3-91b2-467db47491bc-kube-api-access-4vzrc" (OuterVolumeSpecName: "kube-api-access-4vzrc") pod "bf537059-3c80-40f3-91b2-467db47491bc" (UID: "bf537059-3c80-40f3-91b2-467db47491bc"). InnerVolumeSpecName "kube-api-access-4vzrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:52:48.339727 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.339685    2576 generic.go:356] "Generic (PLEG): container finished" podID="bf537059-3c80-40f3-91b2-467db47491bc" containerID="5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d" exitCode=0
Sep 11 16:52:48.339887 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.339768    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"
Sep 11 16:52:48.339887 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.339767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" event={"ID":"bf537059-3c80-40f3-91b2-467db47491bc","Type":"ContainerDied","Data":"5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d"}
Sep 11 16:52:48.339887 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.339851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9" event={"ID":"bf537059-3c80-40f3-91b2-467db47491bc","Type":"ContainerDied","Data":"6aaef21db7460e2d3340aac35a8cff80f1dabfbaf3141516cda29e5482e42492"}
Sep 11 16:52:48.339887 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.339877    2576 scope.go:117] "RemoveContainer" containerID="a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3"
Sep 11 16:52:48.348379 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.348364    2576 scope.go:117] "RemoveContainer" containerID="5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d"
Sep 11 16:52:48.355634 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.355616    2576 scope.go:117] "RemoveContainer" containerID="86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a"
Sep 11 16:52:48.362447 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.362433    2576 scope.go:117] "RemoveContainer" containerID="a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3"
Sep 11 16:52:48.362860 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:52:48.362832    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3\": container with ID starting with a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3 not found: ID does not exist" containerID="a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3"
Sep 11 16:52:48.363056 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.362871    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3"} err="failed to get container status \"a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3\": rpc error: code = NotFound desc = could not find container \"a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3\": container with ID starting with a4dad4f2e6b742d9f5a07fc99e936321ae2813f87e45b627a7c56f35a67fa1c3 not found: ID does not exist"
Sep 11 16:52:48.363056 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.362895    2576 scope.go:117] "RemoveContainer" containerID="5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d"
Sep 11 16:52:48.363243 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:52:48.363220    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d\": container with ID starting with 5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d not found: ID does not exist" containerID="5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d"
Sep 11 16:52:48.363307 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.363253    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d"} err="failed to get container status \"5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d\": rpc error: code = NotFound desc = could not find container \"5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d\": container with ID starting with 5ee069a1f03988d8ec69609484521bbf2a04c12b9a60f84d9aaec31ea5fa0a7d not found: ID does not exist"
Sep 11 16:52:48.363307 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.363276    2576 scope.go:117] "RemoveContainer" containerID="86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a"
Sep 11 16:52:48.363591 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:52:48.363570    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a\": container with ID starting with 86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a not found: ID does not exist" containerID="86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a"
Sep 11 16:52:48.363714 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.363690    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a"} err="failed to get container status \"86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a\": rpc error: code = NotFound desc = could not find container \"86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a\": container with ID starting with 86609a2aeb8f6925f8779d921dc0a8ef1bf65113c3a31f5db97b2e6481764c7a not found: ID does not exist"
Sep 11 16:52:48.364855 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.364837    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"]
Sep 11 16:52:48.370192 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.370169    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9"]
Sep 11 16:52:48.372112 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.372095    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vzrc\" (UniqueName: \"kubernetes.io/projected/bf537059-3c80-40f3-91b2-467db47491bc-kube-api-access-4vzrc\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:48.372172 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.372116    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:48.372172 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.372125    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf537059-3c80-40f3-91b2-467db47491bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:48.372172 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.372133    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf537059-3c80-40f3-91b2-467db47491bc-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:48.372172 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.372141    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf537059-3c80-40f3-91b2-467db47491bc-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:52:48.603505 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:48.603428    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf537059-3c80-40f3-91b2-467db47491bc" path="/var/lib/kubelet/pods/bf537059-3c80-40f3-91b2-467db47491bc/volumes"
Sep 11 16:52:50.507963 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:50.507911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:50.715919 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:50.715879    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:53.300178 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:53.300133    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 11 16:52:53.300568 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:52:53.300139    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:53:03.300973 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:03.300913    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:53:03.319076 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:03.300913    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 11 16:53:13.300279 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:13.300239    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:53:13.300279 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:13.300263    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 11 16:53:23.300480 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:23.300437    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:53:23.300964 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:23.300447    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 11 16:53:33.300591 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:33.300548    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:53:33.301083 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:33.300548    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 11 16:53:43.077041 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:43.077001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:43.175228 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:43.175194    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:43.300007 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:43.299968    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:53:43.300198 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:43.300109    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 11 16:53:52.602824 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:52.602793    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:53:53.301101 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:53.301072    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:53:53.469672 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:53.469635    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"]
Sep 11 16:53:54.609693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:54.609657    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container" containerID="cri-o://8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254" gracePeriod=30
Sep 11 16:53:54.609693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:54.609674    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kube-rbac-proxy" containerID="cri-o://69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912" gracePeriod=30
Sep 11 16:53:55.616050 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:55.616014    2576 generic.go:356] "Generic (PLEG): container finished" podID="f4634433-246b-452d-a6bc-84ef26b98359" containerID="69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912" exitCode=2
Sep 11 16:53:55.616463 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:55.616079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" event={"ID":"f4634433-246b-452d-a6bc-84ef26b98359","Type":"ContainerDied","Data":"69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912"}
Sep 11 16:53:58.294823 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:58.294768    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 11 16:53:59.257825 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.257802    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:53:59.322275 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.322245    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-kube-rbac-proxy-sar-config\") pod \"f4634433-246b-452d-a6bc-84ef26b98359\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") "
Sep 11 16:53:59.322275 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.322278    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-cabundle-cert\") pod \"f4634433-246b-452d-a6bc-84ef26b98359\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") "
Sep 11 16:53:59.322687 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.322320    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4634433-246b-452d-a6bc-84ef26b98359-kserve-provision-location\") pod \"f4634433-246b-452d-a6bc-84ef26b98359\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") "
Sep 11 16:53:59.322687 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.322430    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlfjd\" (UniqueName: \"kubernetes.io/projected/f4634433-246b-452d-a6bc-84ef26b98359-kube-api-access-hlfjd\") pod \"f4634433-246b-452d-a6bc-84ef26b98359\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") "
Sep 11 16:53:59.322687 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.322594    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4634433-246b-452d-a6bc-84ef26b98359-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4634433-246b-452d-a6bc-84ef26b98359" (UID: "f4634433-246b-452d-a6bc-84ef26b98359"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:59.322687 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.322658    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4634433-246b-452d-a6bc-84ef26b98359" (UID: "f4634433-246b-452d-a6bc-84ef26b98359"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:53:59.322687 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.322672    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4634433-246b-452d-a6bc-84ef26b98359-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:53:59.322859 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.322671    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4634433-246b-452d-a6bc-84ef26b98359" (UID: "f4634433-246b-452d-a6bc-84ef26b98359"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:53:59.324821 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.324795    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4634433-246b-452d-a6bc-84ef26b98359-kube-api-access-hlfjd" (OuterVolumeSpecName: "kube-api-access-hlfjd") pod "f4634433-246b-452d-a6bc-84ef26b98359" (UID: "f4634433-246b-452d-a6bc-84ef26b98359"). InnerVolumeSpecName "kube-api-access-hlfjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:53:59.423779 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.423694    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4634433-246b-452d-a6bc-84ef26b98359-proxy-tls\") pod \"f4634433-246b-452d-a6bc-84ef26b98359\" (UID: \"f4634433-246b-452d-a6bc-84ef26b98359\") "
Sep 11 16:53:59.423927 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.423823    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlfjd\" (UniqueName: \"kubernetes.io/projected/f4634433-246b-452d-a6bc-84ef26b98359-kube-api-access-hlfjd\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:53:59.423927 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.423835    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:53:59.423927 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.423844    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4634433-246b-452d-a6bc-84ef26b98359-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:53:59.425880 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.425858    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4634433-246b-452d-a6bc-84ef26b98359-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4634433-246b-452d-a6bc-84ef26b98359" (UID: "f4634433-246b-452d-a6bc-84ef26b98359"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:53:59.524217 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.524181    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4634433-246b-452d-a6bc-84ef26b98359-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:53:59.593787 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.593755    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"]
Sep 11 16:53:59.594139 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.594103    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" containerID="cri-o://5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a" gracePeriod=30
Sep 11 16:53:59.594272 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.594142    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kube-rbac-proxy" containerID="cri-o://a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59" gracePeriod=30
Sep 11 16:53:59.637385 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.637240    2576 generic.go:356] "Generic (PLEG): container finished" podID="f4634433-246b-452d-a6bc-84ef26b98359" containerID="8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254" exitCode=0
Sep 11 16:53:59.637570 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.637400    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"
Sep 11 16:53:59.637570 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.637358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" event={"ID":"f4634433-246b-452d-a6bc-84ef26b98359","Type":"ContainerDied","Data":"8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254"}
Sep 11 16:53:59.637570 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.637524    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk" event={"ID":"f4634433-246b-452d-a6bc-84ef26b98359","Type":"ContainerDied","Data":"7d25a9b0a0b76b575f5b3c4455288124d35c740d74da9a198d0d4f92e7356492"}
Sep 11 16:53:59.637570 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.637555    2576 scope.go:117] "RemoveContainer" containerID="69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912"
Sep 11 16:53:59.646782 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.646761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:59.704813 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.704792    2576 scope.go:117] "RemoveContainer" containerID="8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254"
Sep 11 16:53:59.712365 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.712345    2576 scope.go:117] "RemoveContainer" containerID="4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248"
Sep 11 16:53:59.719607 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.719590    2576 scope.go:117] "RemoveContainer" containerID="69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912"
Sep 11 16:53:59.719892 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:53:59.719866    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912\": container with ID starting with 69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912 not found: ID does not exist" containerID="69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912"
Sep 11 16:53:59.719975 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.719902    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912"} err="failed to get container status \"69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912\": rpc error: code = NotFound desc = could not find container \"69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912\": container with ID starting with 69a107e64c2186caf814a3b820aaa6b28392a138606247901dc1ab4c19422912 not found: ID does not exist"
Sep 11 16:53:59.719975 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.719919    2576 scope.go:117] "RemoveContainer" containerID="8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254"
Sep 11 16:53:59.720275 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:53:59.720248    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254\": container with ID starting with 8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254 not found: ID does not exist" containerID="8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254"
Sep 11 16:53:59.720365 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.720276    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254"} err="failed to get container status \"8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254\": rpc error: code = NotFound desc = could not find container \"8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254\": container with ID starting with 8133f67cb5638aac90124e01104fd5e5754fdb45903bb50869cacada43f48254 not found: ID does not exist"
Sep 11 16:53:59.720365 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.720289    2576 scope.go:117] "RemoveContainer" containerID="4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248"
Sep 11 16:53:59.720567 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:53:59.720549    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248\": container with ID starting with 4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248 not found: ID does not exist" containerID="4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248"
Sep 11 16:53:59.720610 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.720579    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248"} err="failed to get container status \"4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248\": rpc error: code = NotFound desc = could not find container \"4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248\": container with ID starting with 4745113830e3dbf21969f5525c507696366b7986cbb7363eb77f496e3990c248 not found: ID does not exist"
Sep 11 16:53:59.730127 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.730106    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"]
Sep 11 16:53:59.740320 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.740298    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk"]
Sep 11 16:53:59.944186 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944154    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"]
Sep 11 16:53:59.944458 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944446    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kube-rbac-proxy"
Sep 11 16:53:59.944502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944461    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kube-rbac-proxy"
Sep 11 16:53:59.944502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944472    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="storage-initializer"
Sep 11 16:53:59.944502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944479    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="storage-initializer"
Sep 11 16:53:59.944502 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944498    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kserve-container"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944505    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kserve-container"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944515    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944520    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944528    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kube-rbac-proxy"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944533    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kube-rbac-proxy"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944541    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944545    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944553    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="storage-initializer"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944558    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="storage-initializer"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944565    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kube-rbac-proxy"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944573    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kube-rbac-proxy"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944584    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="storage-initializer"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944589    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="storage-initializer"
Sep 11 16:53:59.944628 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944633    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kserve-container"
Sep 11 16:53:59.945051 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944644    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kserve-container"
Sep 11 16:53:59.945051 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944654    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4634433-246b-452d-a6bc-84ef26b98359" containerName="kube-rbac-proxy"
Sep 11 16:53:59.945051 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944661    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kube-rbac-proxy"
Sep 11 16:53:59.945051 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944666    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5cecc23-ac05-468d-9081-277fb4618ab5" containerName="kserve-container"
Sep 11 16:53:59.945051 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.944671    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf537059-3c80-40f3-91b2-467db47491bc" containerName="kube-rbac-proxy"
Sep 11 16:53:59.947531 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.947510    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:53:59.950400 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:53:59.950368    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 11 16:54:00.013661 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.013578    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"]
Sep 11 16:54:00.027155 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.027118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.027310 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.027165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49b324b7-f071-4feb-b568-a8304849d84a-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.027310 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.027197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xddcq\" (UniqueName: \"kubernetes.io/projected/49b324b7-f071-4feb-b568-a8304849d84a-kube-api-access-xddcq\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.027310 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.027231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.027310 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.027249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49b324b7-f071-4feb-b568-a8304849d84a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.128069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.128042    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49b324b7-f071-4feb-b568-a8304849d84a-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.128232 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.128080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddcq\" (UniqueName: \"kubernetes.io/projected/49b324b7-f071-4feb-b568-a8304849d84a-kube-api-access-xddcq\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.128232 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.128105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.128232 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.128123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49b324b7-f071-4feb-b568-a8304849d84a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.128340 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.128253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.128578 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.128558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49b324b7-f071-4feb-b568-a8304849d84a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.128844 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.128825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.128904 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.128858    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.130649 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.130623    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49b324b7-f071-4feb-b568-a8304849d84a-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.141477 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.141453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddcq\" (UniqueName: \"kubernetes.io/projected/49b324b7-f071-4feb-b568-a8304849d84a-kube-api-access-xddcq\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-s77ll\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.257698 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.257663    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:00.536759 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.536689    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"]
Sep 11 16:54:00.539235 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:54:00.539209    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b324b7_f071_4feb_b568_a8304849d84a.slice/crio-7a4c69383f01948c70899a550915de0982e41a9bec0c2928f0e8063ccaf779bb WatchSource:0}: Error finding container 7a4c69383f01948c70899a550915de0982e41a9bec0c2928f0e8063ccaf779bb: Status 404 returned error can't find the container with id 7a4c69383f01948c70899a550915de0982e41a9bec0c2928f0e8063ccaf779bb
Sep 11 16:54:00.602525 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.602499    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4634433-246b-452d-a6bc-84ef26b98359" path="/var/lib/kubelet/pods/f4634433-246b-452d-a6bc-84ef26b98359/volumes"
Sep 11 16:54:00.643074 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.643035    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" event={"ID":"49b324b7-f071-4feb-b568-a8304849d84a","Type":"ContainerStarted","Data":"b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc"}
Sep 11 16:54:00.643222 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.643104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" event={"ID":"49b324b7-f071-4feb-b568-a8304849d84a","Type":"ContainerStarted","Data":"7a4c69383f01948c70899a550915de0982e41a9bec0c2928f0e8063ccaf779bb"}
Sep 11 16:54:00.646292 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.646261    2576 generic.go:356] "Generic (PLEG): container finished" podID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerID="a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59" exitCode=2
Sep 11 16:54:00.646403 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:00.646315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" event={"ID":"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0","Type":"ContainerDied","Data":"a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59"}
Sep 11 16:54:03.294376 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:03.294330    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 11 16:54:03.300711 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:03.300681    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 11 16:54:04.442312 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.442288    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:54:04.460125 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.460103    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-proxy-tls\") pod \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") "
Sep 11 16:54:04.460248 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.460149    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-rbac-proxy-sar-config\") pod \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") "
Sep 11 16:54:04.460248 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.460176    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-cabundle-cert\") pod \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") "
Sep 11 16:54:04.460248 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.460205    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp7bf\" (UniqueName: \"kubernetes.io/projected/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-api-access-qp7bf\") pod \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") "
Sep 11 16:54:04.460248 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.460238    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kserve-provision-location\") pod \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\" (UID: \"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0\") "
Sep 11 16:54:04.460541 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.460511    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" (UID: "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:54:04.460666 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.460619    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" (UID: "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:54:04.460666 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.460637    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" (UID: "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:54:04.462386 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.462363    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" (UID: "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:54:04.462462 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.462440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-api-access-qp7bf" (OuterVolumeSpecName: "kube-api-access-qp7bf") pod "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" (UID: "46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0"). InnerVolumeSpecName "kube-api-access-qp7bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:54:04.561814 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.561723    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:54:04.561814 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.561756    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:54:04.561814 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.561768    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:54:04.561814 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.561781    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7bf\" (UniqueName: \"kubernetes.io/projected/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kube-api-access-qp7bf\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:54:04.561814 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.561792    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:54:04.662642 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.662615    2576 generic.go:356] "Generic (PLEG): container finished" podID="49b324b7-f071-4feb-b568-a8304849d84a" containerID="b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc" exitCode=0
Sep 11 16:54:04.662767 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.662686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" event={"ID":"49b324b7-f071-4feb-b568-a8304849d84a","Type":"ContainerDied","Data":"b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc"}
Sep 11 16:54:04.664750 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.664711    2576 generic.go:356] "Generic (PLEG): container finished" podID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerID="5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a" exitCode=0
Sep 11 16:54:04.664808 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.664763    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" event={"ID":"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0","Type":"ContainerDied","Data":"5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a"}
Sep 11 16:54:04.664808 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.664770    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"
Sep 11 16:54:04.664881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.664814    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj" event={"ID":"46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0","Type":"ContainerDied","Data":"72385d5fd559a0d90f5d9622c8342de07badf9555ae1d35044012d6815839868"}
Sep 11 16:54:04.664881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.664841    2576 scope.go:117] "RemoveContainer" containerID="a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59"
Sep 11 16:54:04.674569 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.674550    2576 scope.go:117] "RemoveContainer" containerID="5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a"
Sep 11 16:54:04.683162 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.683139    2576 scope.go:117] "RemoveContainer" containerID="05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f"
Sep 11 16:54:04.697616 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.697596    2576 scope.go:117] "RemoveContainer" containerID="a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59"
Sep 11 16:54:04.697954 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:54:04.697911    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59\": container with ID starting with a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59 not found: ID does not exist" containerID="a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59"
Sep 11 16:54:04.698020 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.697975    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59"} err="failed to get container status \"a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59\": rpc error: code = NotFound desc = could not find container \"a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59\": container with ID starting with a5e9e416345675d43548fa6bc90042d0f922f6531e9b742065830bcc3caa6d59 not found: ID does not exist"
Sep 11 16:54:04.698020 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.697997    2576 scope.go:117] "RemoveContainer" containerID="5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a"
Sep 11 16:54:04.698286 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:54:04.698261    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a\": container with ID starting with 5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a not found: ID does not exist" containerID="5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a"
Sep 11 16:54:04.698335 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.698296    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a"} err="failed to get container status \"5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a\": rpc error: code = NotFound desc = could not find container \"5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a\": container with ID starting with 5840d40f065fecef2e042b74d2a20769c8a5307ea332b380eec941d2521b6d5a not found: ID does not exist"
Sep 11 16:54:04.698335 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.698319    2576 scope.go:117] "RemoveContainer" containerID="05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f"
Sep 11 16:54:04.698555 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:54:04.698538    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f\": container with ID starting with 05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f not found: ID does not exist" containerID="05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f"
Sep 11 16:54:04.698609 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.698563    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f"} err="failed to get container status \"05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f\": rpc error: code = NotFound desc = could not find container \"05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f\": container with ID starting with 05dd567251ca3296dc4e980b786cd3b58500a4fd2e7c35e99044e7c4ac29fc7f not found: ID does not exist"
Sep 11 16:54:04.704696 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.704673    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"]
Sep 11 16:54:04.711462 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:04.711397    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj"]
Sep 11 16:54:05.671513 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:05.671434    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" event={"ID":"49b324b7-f071-4feb-b568-a8304849d84a","Type":"ContainerStarted","Data":"4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479"}
Sep 11 16:54:05.671513 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:05.671482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" event={"ID":"49b324b7-f071-4feb-b568-a8304849d84a","Type":"ContainerStarted","Data":"5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386"}
Sep 11 16:54:05.671898 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:05.671665    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:05.706482 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:05.706255    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podStartSLOduration=2.707007462 podStartE2EDuration="6.706234597s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.999227135s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:54:05.705986621 +0000 UTC m=+2239.615787784" watchObservedRunningTime="2026-09-11 16:54:05.706234597 +0000 UTC m=+2239.616035762"
Sep 11 16:54:06.603043 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:06.603007    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" path="/var/lib/kubelet/pods/46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0/volumes"
Sep 11 16:54:06.675629 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:06.675596    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:06.676796 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:06.676767    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:54:07.679526 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:07.679442    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:54:12.684607 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:12.684577    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:54:12.685123 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:12.685094    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:54:20.694528 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:20.694490    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:22.685111 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:22.685074    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:54:25.923473 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:25.923432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:26.021870 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:26.021833    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:32.685301 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:32.685256    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:54:42.685388 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:42.685347    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:54:52.685371 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:54:52.685336    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:55:02.685624 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:02.685574    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:55:12.685167 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:12.685122    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:55:16.194536 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:16.194495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:21.133496 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:21.133462    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:21.252277 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:21.252241    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:22.311741 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:22.311695    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:22.686062 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:22.685983    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:55:29.767550 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:29.767505    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"]
Sep 11 16:55:29.768089 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:29.767759    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" containerID="cri-o://5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386" gracePeriod=30
Sep 11 16:55:29.768089 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:29.767830    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kube-rbac-proxy" containerID="cri-o://4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479" gracePeriod=30
Sep 11 16:55:29.791351 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:29.791318    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:29.989157 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:29.989116    2576 generic.go:356] "Generic (PLEG): container finished" podID="49b324b7-f071-4feb-b568-a8304849d84a" containerID="4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479" exitCode=2
Sep 11 16:55:29.989157 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:29.989154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" event={"ID":"49b324b7-f071-4feb-b568-a8304849d84a","Type":"ContainerDied","Data":"4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479"}
Sep 11 16:55:32.680097 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:32.680052    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 11 16:55:32.685595 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:32.685567    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 11 16:55:34.602820 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.602791    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:55:34.640634 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.640602    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-kube-rbac-proxy-sar-config\") pod \"49b324b7-f071-4feb-b568-a8304849d84a\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") "
Sep 11 16:55:34.640800 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.640650    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xddcq\" (UniqueName: \"kubernetes.io/projected/49b324b7-f071-4feb-b568-a8304849d84a-kube-api-access-xddcq\") pod \"49b324b7-f071-4feb-b568-a8304849d84a\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") "
Sep 11 16:55:34.640800 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.640677    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49b324b7-f071-4feb-b568-a8304849d84a-kserve-provision-location\") pod \"49b324b7-f071-4feb-b568-a8304849d84a\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") "
Sep 11 16:55:34.640800 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.640738    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49b324b7-f071-4feb-b568-a8304849d84a-proxy-tls\") pod \"49b324b7-f071-4feb-b568-a8304849d84a\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") "
Sep 11 16:55:34.640800 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.640795    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-cabundle-cert\") pod \"49b324b7-f071-4feb-b568-a8304849d84a\" (UID: \"49b324b7-f071-4feb-b568-a8304849d84a\") "
Sep 11 16:55:34.641047 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.640985    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49b324b7-f071-4feb-b568-a8304849d84a" (UID: "49b324b7-f071-4feb-b568-a8304849d84a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:55:34.641109 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.641038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b324b7-f071-4feb-b568-a8304849d84a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49b324b7-f071-4feb-b568-a8304849d84a" (UID: "49b324b7-f071-4feb-b568-a8304849d84a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:34.641355 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.641288    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49b324b7-f071-4feb-b568-a8304849d84a" (UID: "49b324b7-f071-4feb-b568-a8304849d84a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:55:34.643057 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.643036    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b324b7-f071-4feb-b568-a8304849d84a-kube-api-access-xddcq" (OuterVolumeSpecName: "kube-api-access-xddcq") pod "49b324b7-f071-4feb-b568-a8304849d84a" (UID: "49b324b7-f071-4feb-b568-a8304849d84a"). InnerVolumeSpecName "kube-api-access-xddcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:55:34.643162 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.643067    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b324b7-f071-4feb-b568-a8304849d84a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49b324b7-f071-4feb-b568-a8304849d84a" (UID: "49b324b7-f071-4feb-b568-a8304849d84a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:55:34.741425 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.741392    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:55:34.741425 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.741418    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49b324b7-f071-4feb-b568-a8304849d84a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:55:34.741425 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.741429    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xddcq\" (UniqueName: \"kubernetes.io/projected/49b324b7-f071-4feb-b568-a8304849d84a-kube-api-access-xddcq\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:55:34.741662 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.741440    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49b324b7-f071-4feb-b568-a8304849d84a-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:55:34.741662 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:34.741450    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49b324b7-f071-4feb-b568-a8304849d84a-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:55:35.011918 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.011880    2576 generic.go:356] "Generic (PLEG): container finished" podID="49b324b7-f071-4feb-b568-a8304849d84a" containerID="5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386" exitCode=0
Sep 11 16:55:35.012148 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.011922    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" event={"ID":"49b324b7-f071-4feb-b568-a8304849d84a","Type":"ContainerDied","Data":"5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386"}
Sep 11 16:55:35.012148 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.011959    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"
Sep 11 16:55:35.012148 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.012004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll" event={"ID":"49b324b7-f071-4feb-b568-a8304849d84a","Type":"ContainerDied","Data":"7a4c69383f01948c70899a550915de0982e41a9bec0c2928f0e8063ccaf779bb"}
Sep 11 16:55:35.012148 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.012041    2576 scope.go:117] "RemoveContainer" containerID="4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479"
Sep 11 16:55:35.021334 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.021313    2576 scope.go:117] "RemoveContainer" containerID="5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386"
Sep 11 16:55:35.027438 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.027393    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:35.028922 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.028905    2576 scope.go:117] "RemoveContainer" containerID="b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc"
Sep 11 16:55:35.036445 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.036422    2576 scope.go:117] "RemoveContainer" containerID="4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479"
Sep 11 16:55:35.036724 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:55:35.036705    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479\": container with ID starting with 4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479 not found: ID does not exist" containerID="4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479"
Sep 11 16:55:35.036774 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.036733    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479"} err="failed to get container status \"4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479\": rpc error: code = NotFound desc = could not find container \"4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479\": container with ID starting with 4390dec8936dac99a6f23267a13c6a776f57b74988f9b17aa84bd2d630147479 not found: ID does not exist"
Sep 11 16:55:35.036774 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.036751    2576 scope.go:117] "RemoveContainer" containerID="5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386"
Sep 11 16:55:35.037013 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:55:35.036997    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386\": container with ID starting with 5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386 not found: ID does not exist" containerID="5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386"
Sep 11 16:55:35.037064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.037018    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386"} err="failed to get container status \"5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386\": rpc error: code = NotFound desc = could not find container \"5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386\": container with ID starting with 5892f073711e44f88a0f107b6376bf8fa25c176f60386ed0f95739b60218d386 not found: ID does not exist"
Sep 11 16:55:35.037064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.037031    2576 scope.go:117] "RemoveContainer" containerID="b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc"
Sep 11 16:55:35.037242 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:55:35.037226    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc\": container with ID starting with b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc not found: ID does not exist" containerID="b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc"
Sep 11 16:55:35.037286 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.037250    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc"} err="failed to get container status \"b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc\": rpc error: code = NotFound desc = could not find container \"b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc\": container with ID starting with b77028524598bb1aa3c70f6f23f0889ba9be957e4e588b20fe33bf88cd34d0dc not found: ID does not exist"
Sep 11 16:55:35.047295 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.047266    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"]
Sep 11 16:55:35.053515 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.053492    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll"]
Sep 11 16:55:35.107560 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107522    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"]
Sep 11 16:55:35.107835 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107822    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="storage-initializer"
Sep 11 16:55:35.107835 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107836    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="storage-initializer"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107850    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107857    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107867    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107873    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107881    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kube-rbac-proxy"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107886    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kube-rbac-proxy"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107895    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="storage-initializer"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107900    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="storage-initializer"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107907    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kube-rbac-proxy"
Sep 11 16:55:35.107958 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107912    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kube-rbac-proxy"
Sep 11 16:55:35.108315 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107987    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kserve-container"
Sep 11 16:55:35.108315 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.107995    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kube-rbac-proxy"
Sep 11 16:55:35.108315 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.108002    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ff62d5-bd3b-4727-b6c3-cae2cf88d4b0" containerName="kube-rbac-proxy"
Sep 11 16:55:35.108315 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.108009    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b324b7-f071-4feb-b568-a8304849d84a" containerName="kserve-container"
Sep 11 16:55:35.110751 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.110734    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.115683 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.115661    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 11 16:55:35.115845 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.115766    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:55:35.115845 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.115834    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:55:35.116254 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.116240    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 11 16:55:35.116459 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.116448    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:55:35.117387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.117344    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:55:35.117508 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.117449    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:55:35.118784 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.118761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:35.128783 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.128756    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"]
Sep 11 16:55:35.144711 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.144685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6km\" (UniqueName: \"kubernetes.io/projected/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-api-access-rc6km\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.144857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.144723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16bf4159-7f62-4984-a4d7-8c2682f81bbe-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.144857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.144754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.144857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.144812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.144857 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.144851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.192402 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.192377    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"]
Sep 11 16:55:35.194485 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.194470    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.198235 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.198213    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 11 16:55:35.210275 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.210254    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"]
Sep 11 16:55:35.246134 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.246264 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.246264 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.246264 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.246264 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfxf2\" (UniqueName: \"kubernetes.io/projected/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-api-access-zfxf2\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.246264 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.246623 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6km\" (UniqueName: \"kubernetes.io/projected/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-api-access-rc6km\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.246623 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16bf4159-7f62-4984-a4d7-8c2682f81bbe-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.246623 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.246623 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.246843 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246777    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.246843 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246786    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.246976 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.246917    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.248882 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.248848    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16bf4159-7f62-4984-a4d7-8c2682f81bbe-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.262482 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.262420    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6km\" (UniqueName: \"kubernetes.io/projected/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-api-access-rc6km\") pod \"isvc-canary-stop-v2-predictor-596664c988-bsz9b\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.268582 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.268559    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:35.347452 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.347427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfxf2\" (UniqueName: \"kubernetes.io/projected/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-api-access-zfxf2\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.347593 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.347463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.347593 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.347522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.347593 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.347548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.347593 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.347578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.347962 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:55:35.347920    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 11 16:55:35.348081 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:55:35.348022    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-proxy-tls podName:26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7 nodeName:}" failed. No retries permitted until 2026-09-11 16:55:35.848001057 +0000 UTC m=+2329.757802203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-proxy-tls") pod "isvc-canary-stop-predictor-5cdcf6545-kqh6q" (UID: "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 11 16:55:35.348081 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.348073    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.348308 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.348289    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.348347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.348297    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.363660 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.363635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfxf2\" (UniqueName: \"kubernetes.io/projected/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-api-access-zfxf2\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.421434 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.421403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:35.588787 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.588707    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:35.697638 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.697614    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"]
Sep 11 16:55:35.699875 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:55:35.699846    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bf4159_7f62_4984_a4d7_8c2682f81bbe.slice/crio-3066ac72134e084c0ad29c13affae2f1576f51abca7666e41095cbfa4d3f9be1 WatchSource:0}: Error finding container 3066ac72134e084c0ad29c13affae2f1576f51abca7666e41095cbfa4d3f9be1: Status 404 returned error can't find the container with id 3066ac72134e084c0ad29c13affae2f1576f51abca7666e41095cbfa4d3f9be1
Sep 11 16:55:35.851768 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.851680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:35.854384 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:35.854359    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-kqh6q\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:36.018411 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:36.018375    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" event={"ID":"16bf4159-7f62-4984-a4d7-8c2682f81bbe","Type":"ContainerStarted","Data":"cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312"}
Sep 11 16:55:36.018607 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:36.018439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" event={"ID":"16bf4159-7f62-4984-a4d7-8c2682f81bbe","Type":"ContainerStarted","Data":"3066ac72134e084c0ad29c13affae2f1576f51abca7666e41095cbfa4d3f9be1"}
Sep 11 16:55:36.104348 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:36.104257    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:36.386075 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:36.386051    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"]
Sep 11 16:55:36.388144 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:55:36.388114    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26df51ea_4b4d_46c6_8ec3_2e29e04a5ef7.slice/crio-c6837d7d28dadf7eaa7e0b86703a485897953c16d7a513560d94b4b8066fa269 WatchSource:0}: Error finding container c6837d7d28dadf7eaa7e0b86703a485897953c16d7a513560d94b4b8066fa269: Status 404 returned error can't find the container with id c6837d7d28dadf7eaa7e0b86703a485897953c16d7a513560d94b4b8066fa269
Sep 11 16:55:36.602556 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:36.602522    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b324b7-f071-4feb-b568-a8304849d84a" path="/var/lib/kubelet/pods/49b324b7-f071-4feb-b568-a8304849d84a/volumes"
Sep 11 16:55:37.023144 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:37.023107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" event={"ID":"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7","Type":"ContainerStarted","Data":"cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536"}
Sep 11 16:55:37.023537 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:37.023187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" event={"ID":"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7","Type":"ContainerStarted","Data":"c6837d7d28dadf7eaa7e0b86703a485897953c16d7a513560d94b4b8066fa269"}
Sep 11 16:55:40.035988 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:40.035949    2576 generic.go:356] "Generic (PLEG): container finished" podID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerID="cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312" exitCode=0
Sep 11 16:55:40.036360 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:40.036003    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" event={"ID":"16bf4159-7f62-4984-a4d7-8c2682f81bbe","Type":"ContainerDied","Data":"cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312"}
Sep 11 16:55:41.041811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:41.041775    2576 generic.go:356] "Generic (PLEG): container finished" podID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerID="cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536" exitCode=0
Sep 11 16:55:41.042259 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:41.041849    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" event={"ID":"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7","Type":"ContainerDied","Data":"cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536"}
Sep 11 16:55:41.049723 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:41.049692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" event={"ID":"16bf4159-7f62-4984-a4d7-8c2682f81bbe","Type":"ContainerStarted","Data":"18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb"}
Sep 11 16:55:41.049830 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:41.049777    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" event={"ID":"16bf4159-7f62-4984-a4d7-8c2682f81bbe","Type":"ContainerStarted","Data":"78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127"}
Sep 11 16:55:41.050960 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:41.050925    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:41.099391 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:41.099348    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podStartSLOduration=2.1831127 podStartE2EDuration="6.099332176s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.916219476s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:41.098117985 +0000 UTC m=+2335.007919148" watchObservedRunningTime="2026-09-11 16:55:41.099332176 +0000 UTC m=+2335.009133337"
Sep 11 16:55:42.055953 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:42.055897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" event={"ID":"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7","Type":"ContainerStarted","Data":"f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4"}
Sep 11 16:55:42.056376 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:42.055983    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" event={"ID":"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7","Type":"ContainerStarted","Data":"1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a"}
Sep 11 16:55:42.056376 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:42.056222    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:42.056544 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:42.056520    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:42.056686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:42.056667    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:42.057493 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:42.057471    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:55:42.057711 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:42.057685    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:55:42.079358 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:42.079313    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podStartSLOduration=3.137634423 podStartE2EDuration="7.079297734s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.941663311s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:42.078405188 +0000 UTC m=+2335.988206350" watchObservedRunningTime="2026-09-11 16:55:42.079297734 +0000 UTC m=+2335.989098896"
Sep 11 16:55:43.062259 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:43.062215    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:55:43.062659 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:43.062279    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:55:44.065978 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:44.065912    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:55:48.067027 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:48.066994    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:55:48.067484 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:48.067459    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:55:49.070283 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:49.070253    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:55:49.070778 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:49.070754    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:55:58.067668 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:58.067626    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:55:59.071072 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:55:59.071034    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:56:08.068431 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:08.068395    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:56:09.071381 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:09.071341    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:56:11.231347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:11.231306    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:16.449083 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.449050    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:16.457082 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.457052    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"]
Sep 11 16:56:16.461827 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.461810    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.466023 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.466008    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 11 16:56:16.466125 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.466108    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 11 16:56:16.479641 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.479617    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"]
Sep 11 16:56:16.569081 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.569037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8dh\" (UniqueName: \"kubernetes.io/projected/19716389-687a-4375-9153-09797b4e8ceb-kube-api-access-2b8dh\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.569081 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.569081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19716389-687a-4375-9153-09797b4e8ceb-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.569295 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.569102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.569295 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.569176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.569295 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.569228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19716389-687a-4375-9153-09797b4e8ceb-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.670170 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.670137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8dh\" (UniqueName: \"kubernetes.io/projected/19716389-687a-4375-9153-09797b4e8ceb-kube-api-access-2b8dh\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.670355 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.670187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19716389-687a-4375-9153-09797b4e8ceb-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.670408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.670373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.670408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.670401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.670515 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.670433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19716389-687a-4375-9153-09797b4e8ceb-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.670754 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.670732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19716389-687a-4375-9153-09797b4e8ceb-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.671126 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.671100    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.671221 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.671105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.673112 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.673095    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19716389-687a-4375-9153-09797b4e8ceb-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.680176 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.680153    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8dh\" (UniqueName: \"kubernetes.io/projected/19716389-687a-4375-9153-09797b4e8ceb-kube-api-access-2b8dh\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-v47xd\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:16.699693 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.699618    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:16.771485 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:16.771444    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:17.055412 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:17.055386    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"]
Sep 11 16:56:17.057144 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:56:17.057117    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19716389_687a_4375_9153_09797b4e8ceb.slice/crio-df582266f2d218f4620da8735c0358cf3b37fa3ede76151f547a5fbbc1b27fd6 WatchSource:0}: Error finding container df582266f2d218f4620da8735c0358cf3b37fa3ede76151f547a5fbbc1b27fd6: Status 404 returned error can't find the container with id df582266f2d218f4620da8735c0358cf3b37fa3ede76151f547a5fbbc1b27fd6
Sep 11 16:56:17.194431 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:17.194389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" event={"ID":"19716389-687a-4375-9153-09797b4e8ceb","Type":"ContainerStarted","Data":"1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca"}
Sep 11 16:56:17.194639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:17.194459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" event={"ID":"19716389-687a-4375-9153-09797b4e8ceb","Type":"ContainerStarted","Data":"df582266f2d218f4620da8735c0358cf3b37fa3ede76151f547a5fbbc1b27fd6"}
Sep 11 16:56:18.067776 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:18.067732    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:56:19.071357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:19.071317    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:56:21.212367 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:21.212332    2576 generic.go:356] "Generic (PLEG): container finished" podID="19716389-687a-4375-9153-09797b4e8ceb" containerID="1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca" exitCode=0
Sep 11 16:56:21.212755 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:21.212406    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" event={"ID":"19716389-687a-4375-9153-09797b4e8ceb","Type":"ContainerDied","Data":"1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca"}
Sep 11 16:56:21.213712 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:21.213698    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:56:28.068271 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:28.068229    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:56:29.071069 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:29.071033    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:56:38.068082 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:38.068037    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:56:39.070926 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:39.070748    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:56:40.299293 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:40.299259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" event={"ID":"19716389-687a-4375-9153-09797b4e8ceb","Type":"ContainerStarted","Data":"33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803"}
Sep 11 16:56:40.299654 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:40.299319    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" event={"ID":"19716389-687a-4375-9153-09797b4e8ceb","Type":"ContainerStarted","Data":"8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58"}
Sep 11 16:56:40.299654 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:40.299500    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:40.320662 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:40.320610    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podStartSLOduration=2.227058158 podStartE2EDuration="24.320595001s" totalImagesPullingTime="18.40705308s" totalInitContainerRuntime="3.686483763s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:56:40.320051206 +0000 UTC m=+2394.229852368" watchObservedRunningTime="2026-09-11 16:56:40.320595001 +0000 UTC m=+2394.230396164"
Sep 11 16:56:41.308205 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:41.308174    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:41.309614 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:41.309576    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 11 16:56:42.311991 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:42.311952    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 11 16:56:47.317088 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:47.317053    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:56:47.317705 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:47.317679    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 11 16:56:48.068427 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:48.068390    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:56:49.070729 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:49.070690    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:56:57.318539 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:57.318491    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 11 16:56:58.067738 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:58.067697    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 11 16:56:58.977955 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:58.977912    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:59.047596 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:59.047557    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:59.071503 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:56:59.071473    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:57:07.318398 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:07.318357    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 11 16:57:08.068099 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:08.068066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:57:08.241083 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:08.241046    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"]
Sep 11 16:57:08.241357 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:08.241329    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" containerID="cri-o://1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a" gracePeriod=30
Sep 11 16:57:08.241467 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:08.241417    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kube-rbac-proxy" containerID="cri-o://f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4" gracePeriod=30
Sep 11 16:57:08.417033 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:08.416928    2576 generic.go:356] "Generic (PLEG): container finished" podID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerID="f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4" exitCode=2
Sep 11 16:57:08.417033 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:08.417009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" event={"ID":"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7","Type":"ContainerDied","Data":"f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4"}
Sep 11 16:57:09.066504 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:09.066461    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 11 16:57:09.071485 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:09.071464    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 11 16:57:10.030214 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:10.030176    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"]
Sep 11 16:57:10.030639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:10.030385    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container" containerID="cri-o://78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127" gracePeriod=30
Sep 11 16:57:10.030639 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:10.030475    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kube-rbac-proxy" containerID="cri-o://18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb" gracePeriod=30
Sep 11 16:57:10.035494 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:10.035446    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:10.102619 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:10.102583    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:10.426891 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:10.426811    2576 generic.go:356] "Generic (PLEG): container finished" podID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerID="18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb" exitCode=2
Sep 11 16:57:10.426891 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:10.426845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" event={"ID":"16bf4159-7f62-4984-a4d7-8c2682f81bbe","Type":"ContainerDied","Data":"18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb"}
Sep 11 16:57:13.062465 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.062418    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 11 16:57:13.276284 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.276260    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:57:13.336243 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.336169    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-cabundle-cert\") pod \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") "
Sep 11 16:57:13.336243 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.336222    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfxf2\" (UniqueName: \"kubernetes.io/projected/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-api-access-zfxf2\") pod \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") "
Sep 11 16:57:13.336471 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.336257    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-rbac-proxy-sar-config\") pod \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") "
Sep 11 16:57:13.336471 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.336295    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kserve-provision-location\") pod \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") "
Sep 11 16:57:13.336471 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.336319    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-proxy-tls\") pod \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\" (UID: \"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7\") "
Sep 11 16:57:13.336634 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.336493    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" (UID: "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:57:13.336686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.336630    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" (UID: "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:13.336686 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.336659    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" (UID: "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:57:13.338511 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.338487    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-api-access-zfxf2" (OuterVolumeSpecName: "kube-api-access-zfxf2") pod "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" (UID: "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7"). InnerVolumeSpecName "kube-api-access-zfxf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:57:13.338597 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.338576    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" (UID: "26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:57:13.437478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.437450    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:13.437478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.437472    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:13.437478 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.437482    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:13.437705 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.437493    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:13.437705 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.437501    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfxf2\" (UniqueName: \"kubernetes.io/projected/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7-kube-api-access-zfxf2\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:13.440149 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.440119    2576 generic.go:356] "Generic (PLEG): container finished" podID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerID="1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a" exitCode=0
Sep 11 16:57:13.440263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.440186    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"
Sep 11 16:57:13.440263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.440193    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" event={"ID":"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7","Type":"ContainerDied","Data":"1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a"}
Sep 11 16:57:13.440263 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.440251    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q" event={"ID":"26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7","Type":"ContainerDied","Data":"c6837d7d28dadf7eaa7e0b86703a485897953c16d7a513560d94b4b8066fa269"}
Sep 11 16:57:13.440366 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.440277    2576 scope.go:117] "RemoveContainer" containerID="f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4"
Sep 11 16:57:13.449120 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.449106    2576 scope.go:117] "RemoveContainer" containerID="1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a"
Sep 11 16:57:13.456337 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.456320    2576 scope.go:117] "RemoveContainer" containerID="cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536"
Sep 11 16:57:13.463177 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.463161    2576 scope.go:117] "RemoveContainer" containerID="f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4"
Sep 11 16:57:13.463438 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:57:13.463420    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4\": container with ID starting with f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4 not found: ID does not exist" containerID="f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4"
Sep 11 16:57:13.463506 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.463446    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4"} err="failed to get container status \"f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4\": rpc error: code = NotFound desc = could not find container \"f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4\": container with ID starting with f3a174661e1e302dabf51ecbf696345e29da1317e266c84b5d228f3c479959b4 not found: ID does not exist"
Sep 11 16:57:13.463506 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.463463    2576 scope.go:117] "RemoveContainer" containerID="1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a"
Sep 11 16:57:13.463694 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:57:13.463677    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a\": container with ID starting with 1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a not found: ID does not exist" containerID="1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a"
Sep 11 16:57:13.463736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.463701    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a"} err="failed to get container status \"1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a\": rpc error: code = NotFound desc = could not find container \"1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a\": container with ID starting with 1abb06c29e617960ab81bf3fbec743d955ff1b2f655dfdec1f624307682d5b4a not found: ID does not exist"
Sep 11 16:57:13.463736 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.463717    2576 scope.go:117] "RemoveContainer" containerID="cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536"
Sep 11 16:57:13.463971 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:57:13.463953    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536\": container with ID starting with cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536 not found: ID does not exist" containerID="cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536"
Sep 11 16:57:13.464017 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.463977    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536"} err="failed to get container status \"cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536\": rpc error: code = NotFound desc = could not find container \"cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536\": container with ID starting with cc57fe7894860a3579077254486a5eb9c704d95f518868233e9a326979b0b536 not found: ID does not exist"
Sep 11 16:57:13.473631 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.473610    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"]
Sep 11 16:57:13.478643 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:13.478623    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q"]
Sep 11 16:57:14.603412 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:14.603378    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" path="/var/lib/kubelet/pods/26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7/volumes"
Sep 11 16:57:14.972868 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:14.972843    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:57:15.050655 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.050627    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-rbac-proxy-sar-config\") pod \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") "
Sep 11 16:57:15.050838 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.050695    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16bf4159-7f62-4984-a4d7-8c2682f81bbe-proxy-tls\") pod \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") "
Sep 11 16:57:15.050838 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.050750    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kserve-provision-location\") pod \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") "
Sep 11 16:57:15.050838 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.050782    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc6km\" (UniqueName: \"kubernetes.io/projected/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-api-access-rc6km\") pod \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") "
Sep 11 16:57:15.050838 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.050834    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-cabundle-cert\") pod \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\" (UID: \"16bf4159-7f62-4984-a4d7-8c2682f81bbe\") "
Sep 11 16:57:15.051120 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.051025    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "16bf4159-7f62-4984-a4d7-8c2682f81bbe" (UID: "16bf4159-7f62-4984-a4d7-8c2682f81bbe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:57:15.051190 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.051164    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16bf4159-7f62-4984-a4d7-8c2682f81bbe" (UID: "16bf4159-7f62-4984-a4d7-8c2682f81bbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:15.051261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.051243    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "16bf4159-7f62-4984-a4d7-8c2682f81bbe" (UID: "16bf4159-7f62-4984-a4d7-8c2682f81bbe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:57:15.052891 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.052868    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16bf4159-7f62-4984-a4d7-8c2682f81bbe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16bf4159-7f62-4984-a4d7-8c2682f81bbe" (UID: "16bf4159-7f62-4984-a4d7-8c2682f81bbe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:57:15.053082 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.052987    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-api-access-rc6km" (OuterVolumeSpecName: "kube-api-access-rc6km") pod "16bf4159-7f62-4984-a4d7-8c2682f81bbe" (UID: "16bf4159-7f62-4984-a4d7-8c2682f81bbe"). InnerVolumeSpecName "kube-api-access-rc6km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:57:15.151534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.151460    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.151534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.151486    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16bf4159-7f62-4984-a4d7-8c2682f81bbe-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.151534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.151496    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.151534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.151505    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6km\" (UniqueName: \"kubernetes.io/projected/16bf4159-7f62-4984-a4d7-8c2682f81bbe-kube-api-access-rc6km\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.151534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.151515    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16bf4159-7f62-4984-a4d7-8c2682f81bbe-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.451953 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.451913    2576 generic.go:356] "Generic (PLEG): container finished" podID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerID="78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127" exitCode=0
Sep 11 16:57:15.452125 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.451987    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"
Sep 11 16:57:15.452125 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.451984    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" event={"ID":"16bf4159-7f62-4984-a4d7-8c2682f81bbe","Type":"ContainerDied","Data":"78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127"}
Sep 11 16:57:15.452125 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.452112    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b" event={"ID":"16bf4159-7f62-4984-a4d7-8c2682f81bbe","Type":"ContainerDied","Data":"3066ac72134e084c0ad29c13affae2f1576f51abca7666e41095cbfa4d3f9be1"}
Sep 11 16:57:15.452265 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.452145    2576 scope.go:117] "RemoveContainer" containerID="18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb"
Sep 11 16:57:15.460950 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.460913    2576 scope.go:117] "RemoveContainer" containerID="78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127"
Sep 11 16:57:15.468153 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.468136    2576 scope.go:117] "RemoveContainer" containerID="cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312"
Sep 11 16:57:15.474439 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.474419    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"]
Sep 11 16:57:15.475534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.475514    2576 scope.go:117] "RemoveContainer" containerID="18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb"
Sep 11 16:57:15.475811 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:57:15.475793    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb\": container with ID starting with 18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb not found: ID does not exist" containerID="18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb"
Sep 11 16:57:15.475867 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.475819    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb"} err="failed to get container status \"18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb\": rpc error: code = NotFound desc = could not find container \"18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb\": container with ID starting with 18c13a4b6d8514e524fe923ed689000172b56798c72935ffae5a7c17dbeab0fb not found: ID does not exist"
Sep 11 16:57:15.475867 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.475837    2576 scope.go:117] "RemoveContainer" containerID="78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127"
Sep 11 16:57:15.476110 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:57:15.476094    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127\": container with ID starting with 78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127 not found: ID does not exist" containerID="78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127"
Sep 11 16:57:15.476162 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.476116    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127"} err="failed to get container status \"78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127\": rpc error: code = NotFound desc = could not find container \"78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127\": container with ID starting with 78e67b57b09361b4fddc1a5f664e07d0100150b73127e19108f76f172778c127 not found: ID does not exist"
Sep 11 16:57:15.476162 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.476131    2576 scope.go:117] "RemoveContainer" containerID="cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312"
Sep 11 16:57:15.476364 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:57:15.476348    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312\": container with ID starting with cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312 not found: ID does not exist" containerID="cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312"
Sep 11 16:57:15.476405 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.476369    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312"} err="failed to get container status \"cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312\": rpc error: code = NotFound desc = could not find container \"cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312\": container with ID starting with cd730ec672bf6c39e9272e59da6182d6f8af173324f760ac693a959d43e93312 not found: ID does not exist"
Sep 11 16:57:15.479702 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:15.479684    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b"]
Sep 11 16:57:16.602387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:16.602354    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" path="/var/lib/kubelet/pods/16bf4159-7f62-4984-a4d7-8c2682f81bbe/volumes"
Sep 11 16:57:17.317625 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:17.317591    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 11 16:57:20.329908 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.329873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:20.377099 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377057    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"]
Sep 11 16:57:20.377443 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377425    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="storage-initializer"
Sep 11 16:57:20.377534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377446    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="storage-initializer"
Sep 11 16:57:20.377534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377460    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container"
Sep 11 16:57:20.377534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377467    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container"
Sep 11 16:57:20.377534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377484    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kube-rbac-proxy"
Sep 11 16:57:20.377534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377493    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kube-rbac-proxy"
Sep 11 16:57:20.377534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377508    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="storage-initializer"
Sep 11 16:57:20.377534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377516    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="storage-initializer"
Sep 11 16:57:20.377534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377530    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kube-rbac-proxy"
Sep 11 16:57:20.377534 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377537    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kube-rbac-proxy"
Sep 11 16:57:20.378087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377556    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container"
Sep 11 16:57:20.378087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377564    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container"
Sep 11 16:57:20.378087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377635    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kserve-container"
Sep 11 16:57:20.378087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377648    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kube-rbac-proxy"
Sep 11 16:57:20.378087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377662    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="16bf4159-7f62-4984-a4d7-8c2682f81bbe" containerName="kube-rbac-proxy"
Sep 11 16:57:20.378087 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.377671    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="26df51ea-4b4d-46c6-8ec3-2e29e04a5ef7" containerName="kserve-container"
Sep 11 16:57:20.382411 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.382391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.385161 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.385144    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 11 16:57:20.385397 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.385381    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 11 16:57:20.397345 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.397320    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"]
Sep 11 16:57:20.498248 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.498213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/372dbfef-4ffd-4dda-a47e-4a79dc274084-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.498425 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.498252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/372dbfef-4ffd-4dda-a47e-4a79dc274084-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.498425 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.498281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwsnn\" (UniqueName: \"kubernetes.io/projected/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-api-access-mwsnn\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.498425 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.498328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.498588 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.498435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.599560 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.599464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.599560 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.599508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/372dbfef-4ffd-4dda-a47e-4a79dc274084-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.599560 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.599530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/372dbfef-4ffd-4dda-a47e-4a79dc274084-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.599842 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.599562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwsnn\" (UniqueName: \"kubernetes.io/projected/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-api-access-mwsnn\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.599842 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.599588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.599842 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:57:20.599644    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 11 16:57:20.599842 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:57:20.599720    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372dbfef-4ffd-4dda-a47e-4a79dc274084-proxy-tls podName:372dbfef-4ffd-4dda-a47e-4a79dc274084 nodeName:}" failed. No retries permitted until 2026-09-11 16:57:21.099702288 +0000 UTC m=+2435.009503430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/372dbfef-4ffd-4dda-a47e-4a79dc274084-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-nz97x" (UID: "372dbfef-4ffd-4dda-a47e-4a79dc274084") : secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 11 16:57:20.600072 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.600014    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/372dbfef-4ffd-4dda-a47e-4a79dc274084-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.600233 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.600212    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.600291 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.600233    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.619574 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.619540    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwsnn\" (UniqueName: \"kubernetes.io/projected/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-api-access-mwsnn\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:20.666994 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.666965    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:20.792539 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:20.792506    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:21.102881 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:21.102848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/372dbfef-4ffd-4dda-a47e-4a79dc274084-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:21.105412 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:21.105380    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/372dbfef-4ffd-4dda-a47e-4a79dc274084-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nz97x\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:21.292449 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:21.292408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:21.567829 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:21.567805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"]
Sep 11 16:57:21.569890 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:57:21.569862    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372dbfef_4ffd_4dda_a47e_4a79dc274084.slice/crio-6de5d05f4326d443e6aa5f34d81cc1d155a3198f55e64f60971757516eac3ac4 WatchSource:0}: Error finding container 6de5d05f4326d443e6aa5f34d81cc1d155a3198f55e64f60971757516eac3ac4: Status 404 returned error can't find the container with id 6de5d05f4326d443e6aa5f34d81cc1d155a3198f55e64f60971757516eac3ac4
Sep 11 16:57:22.483261 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:22.483228    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" event={"ID":"372dbfef-4ffd-4dda-a47e-4a79dc274084","Type":"ContainerStarted","Data":"2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666"}
Sep 11 16:57:22.483480 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:22.483293    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" event={"ID":"372dbfef-4ffd-4dda-a47e-4a79dc274084","Type":"ContainerStarted","Data":"6de5d05f4326d443e6aa5f34d81cc1d155a3198f55e64f60971757516eac3ac4"}
Sep 11 16:57:25.498784 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:25.498755    2576 generic.go:356] "Generic (PLEG): container finished" podID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerID="2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666" exitCode=0
Sep 11 16:57:25.499139 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:25.498824    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" event={"ID":"372dbfef-4ffd-4dda-a47e-4a79dc274084","Type":"ContainerDied","Data":"2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666"}
Sep 11 16:57:27.318292 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:27.318242    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 11 16:57:31.832833 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:31.832805    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:57:31.833314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:31.833098    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 16:57:31.839347 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:31.839327    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:57:31.839549 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:31.839530    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 16:57:32.532135 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:32.532103    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" event={"ID":"372dbfef-4ffd-4dda-a47e-4a79dc274084","Type":"ContainerStarted","Data":"3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20"}
Sep 11 16:57:32.532307 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:32.532192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" event={"ID":"372dbfef-4ffd-4dda-a47e-4a79dc274084","Type":"ContainerStarted","Data":"fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e"}
Sep 11 16:57:32.532406 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:32.532391    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:32.554950 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:32.554891    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podStartSLOduration=2.322393279 podStartE2EDuration="12.554863983s" totalImagesPullingTime="6.381098231s" totalInitContainerRuntime="3.851372473s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:57:32.554060409 +0000 UTC m=+2446.463861572" watchObservedRunningTime="2026-09-11 16:57:32.554863983 +0000 UTC m=+2446.464665145"
Sep 11 16:57:33.536094 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:33.536070    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:33.537432 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:33.537410    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:57:34.539532 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:34.539494    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:57:37.318186 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:37.318145    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 11 16:57:39.544462 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:39.544431    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:57:39.544972 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:39.544918    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:57:47.319094 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:47.319064    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:57:49.545716 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:49.545679    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:57:56.532039 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:56.531996    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"]
Sep 11 16:57:56.532606 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:56.532324    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kube-rbac-proxy" containerID="cri-o://33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803" gracePeriod=30
Sep 11 16:57:56.532606 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:56.532308    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" containerID="cri-o://8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58" gracePeriod=30
Sep 11 16:57:56.534315 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:56.534290    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:57.313037 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:57.312991    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 11 16:57:57.318280 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:57.318255    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 11 16:57:57.630575 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:57.630493    2576 generic.go:356] "Generic (PLEG): container finished" podID="19716389-687a-4375-9153-09797b4e8ceb" containerID="33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803" exitCode=2
Sep 11 16:57:57.630575 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:57.630554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" event={"ID":"19716389-687a-4375-9153-09797b4e8ceb","Type":"ContainerDied","Data":"33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803"}
Sep 11 16:57:59.545110 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:57:59.545076    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:58:00.377825 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.377797    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:58:00.414512 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.414482    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19716389-687a-4375-9153-09797b4e8ceb-kserve-provision-location\") pod \"19716389-687a-4375-9153-09797b4e8ceb\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") "
Sep 11 16:58:00.414682 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.414525    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19716389-687a-4375-9153-09797b4e8ceb-proxy-tls\") pod \"19716389-687a-4375-9153-09797b4e8ceb\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") "
Sep 11 16:58:00.414682 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.414561    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-kube-rbac-proxy-sar-config\") pod \"19716389-687a-4375-9153-09797b4e8ceb\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") "
Sep 11 16:58:00.414682 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.414596    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-cabundle-cert\") pod \"19716389-687a-4375-9153-09797b4e8ceb\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") "
Sep 11 16:58:00.414682 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.414634    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b8dh\" (UniqueName: \"kubernetes.io/projected/19716389-687a-4375-9153-09797b4e8ceb-kube-api-access-2b8dh\") pod \"19716389-687a-4375-9153-09797b4e8ceb\" (UID: \"19716389-687a-4375-9153-09797b4e8ceb\") "
Sep 11 16:58:00.414914 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.414794    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19716389-687a-4375-9153-09797b4e8ceb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19716389-687a-4375-9153-09797b4e8ceb" (UID: "19716389-687a-4375-9153-09797b4e8ceb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:00.415024 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.415000    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19716389-687a-4375-9153-09797b4e8ceb" (UID: "19716389-687a-4375-9153-09797b4e8ceb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:58:00.415115 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.415016    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19716389-687a-4375-9153-09797b4e8ceb" (UID: "19716389-687a-4375-9153-09797b4e8ceb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:58:00.416897 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.416876    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19716389-687a-4375-9153-09797b4e8ceb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19716389-687a-4375-9153-09797b4e8ceb" (UID: "19716389-687a-4375-9153-09797b4e8ceb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:58:00.417035 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.416912    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19716389-687a-4375-9153-09797b4e8ceb-kube-api-access-2b8dh" (OuterVolumeSpecName: "kube-api-access-2b8dh") pod "19716389-687a-4375-9153-09797b4e8ceb" (UID: "19716389-687a-4375-9153-09797b4e8ceb"). InnerVolumeSpecName "kube-api-access-2b8dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:58:00.515424 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.515394    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:58:00.515424 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.515419    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19716389-687a-4375-9153-09797b4e8ceb-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:58:00.515424 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.515429    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b8dh\" (UniqueName: \"kubernetes.io/projected/19716389-687a-4375-9153-09797b4e8ceb-kube-api-access-2b8dh\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:58:00.515672 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.515438    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19716389-687a-4375-9153-09797b4e8ceb-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:58:00.515672 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.515448    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19716389-687a-4375-9153-09797b4e8ceb-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:58:00.643498 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.643464    2576 generic.go:356] "Generic (PLEG): container finished" podID="19716389-687a-4375-9153-09797b4e8ceb" containerID="8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58" exitCode=0
Sep 11 16:58:00.643850 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.643525    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"
Sep 11 16:58:00.643850 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.643533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" event={"ID":"19716389-687a-4375-9153-09797b4e8ceb","Type":"ContainerDied","Data":"8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58"}
Sep 11 16:58:00.643850 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.643590    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd" event={"ID":"19716389-687a-4375-9153-09797b4e8ceb","Type":"ContainerDied","Data":"df582266f2d218f4620da8735c0358cf3b37fa3ede76151f547a5fbbc1b27fd6"}
Sep 11 16:58:00.643850 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.643627    2576 scope.go:117] "RemoveContainer" containerID="33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803"
Sep 11 16:58:00.651642 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.651620    2576 scope.go:117] "RemoveContainer" containerID="8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58"
Sep 11 16:58:00.658896 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.658879    2576 scope.go:117] "RemoveContainer" containerID="1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca"
Sep 11 16:58:00.665880 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.665859    2576 scope.go:117] "RemoveContainer" containerID="33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803"
Sep 11 16:58:00.666162 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:58:00.666142    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803\": container with ID starting with 33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803 not found: ID does not exist" containerID="33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803"
Sep 11 16:58:00.666234 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.666169    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803"} err="failed to get container status \"33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803\": rpc error: code = NotFound desc = could not find container \"33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803\": container with ID starting with 33aadc5445b74fea76b591ca274d58877e8c2159be098e3898f872d178786803 not found: ID does not exist"
Sep 11 16:58:00.666234 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.666186    2576 scope.go:117] "RemoveContainer" containerID="8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58"
Sep 11 16:58:00.666436 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:58:00.666421    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58\": container with ID starting with 8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58 not found: ID does not exist" containerID="8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58"
Sep 11 16:58:00.666474 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.666438    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58"} err="failed to get container status \"8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58\": rpc error: code = NotFound desc = could not find container \"8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58\": container with ID starting with 8860f92b147b7055709001c17314dc15d3bb5529ccd3b3d5bac8644d9cb9aa58 not found: ID does not exist"
Sep 11 16:58:00.666474 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.666450    2576 scope.go:117] "RemoveContainer" containerID="1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca"
Sep 11 16:58:00.666674 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:58:00.666658    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca\": container with ID starting with 1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca not found: ID does not exist" containerID="1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca"
Sep 11 16:58:00.666717 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.666680    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca"} err="failed to get container status \"1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca\": rpc error: code = NotFound desc = could not find container \"1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca\": container with ID starting with 1400ff9358d6e4a60b1b7aa7d26427a2ef101d61ec8e5ea25d261a1de97ad7ca not found: ID does not exist"
Sep 11 16:58:00.670756 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.670733    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"]
Sep 11 16:58:00.680615 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:00.680591    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd"]
Sep 11 16:58:01.852291 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:01.852253    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:02.180442 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:02.180361    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:02.602895 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:02.602859    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19716389-687a-4375-9153-09797b4e8ceb" path="/var/lib/kubelet/pods/19716389-687a-4375-9153-09797b4e8ceb/volumes"
Sep 11 16:58:09.545548 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:09.545499    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:58:19.545638 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:19.545593    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:58:29.545107 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:29.545060    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:58:39.545075 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:39.544991    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:58:45.599044 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:45.598997    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:58:55.599805 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:58:55.599773    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:59:01.852094 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:01.852063    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:04.565883 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:04.565852    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"]
Sep 11 16:59:04.566326 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:04.566112    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" containerID="cri-o://fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e" gracePeriod=30
Sep 11 16:59:04.566326 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:04.566148    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kube-rbac-proxy" containerID="cri-o://3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20" gracePeriod=30
Sep 11 16:59:04.582569 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:04.582540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:04.896194 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:04.896115    2576 generic.go:356] "Generic (PLEG): container finished" podID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerID="3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20" exitCode=2
Sep 11 16:59:04.896194 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:04.896164    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" event={"ID":"372dbfef-4ffd-4dda-a47e-4a79dc274084","Type":"ContainerDied","Data":"3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20"}
Sep 11 16:59:05.599408 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:05.599369    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 11 16:59:06.998493 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:06.998454    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:07.057853 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.057819    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"]
Sep 11 16:59:07.058131 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.058110    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container"
Sep 11 16:59:07.058131 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.058125    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container"
Sep 11 16:59:07.058254 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.058145    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="storage-initializer"
Sep 11 16:59:07.058254 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.058158    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="storage-initializer"
Sep 11 16:59:07.058254 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.058169    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kube-rbac-proxy"
Sep 11 16:59:07.058254 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.058174    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kube-rbac-proxy"
Sep 11 16:59:07.058254 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.058221    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kube-rbac-proxy"
Sep 11 16:59:07.058254 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.058230    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19716389-687a-4375-9153-09797b4e8ceb" containerName="kserve-container"
Sep 11 16:59:07.061314 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.061296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.063723 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.063699    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 11 16:59:07.063883 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.063709    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 11 16:59:07.073752 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.073723    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"]
Sep 11 16:59:07.185635 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.185601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.185811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.185660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff81ba77-ad45-40ed-aa9e-4da8391b8492-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.185811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.185696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.185811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.185739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8l6l\" (UniqueName: \"kubernetes.io/projected/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-api-access-n8l6l\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.185811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.185771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.215329 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.215295    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:07.286989 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.286872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.286989 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.286923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff81ba77-ad45-40ed-aa9e-4da8391b8492-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.286989 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.286965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.287260 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.287074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8l6l\" (UniqueName: \"kubernetes.io/projected/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-api-access-n8l6l\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.287260 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.287116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.287500 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.287472    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.287713 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.287694    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.287790 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.287743    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.289557 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.289537    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff81ba77-ad45-40ed-aa9e-4da8391b8492-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.299209 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.299187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8l6l\" (UniqueName: \"kubernetes.io/projected/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-api-access-n8l6l\") pod \"isvc-xgboost-v2-predictor-fcc668bc-nmnxm\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.371875 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.371836    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:07.647341 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.647191    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"]
Sep 11 16:59:07.649450 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:59:07.649423    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff81ba77_ad45_40ed_aa9e_4da8391b8492.slice/crio-e08ba545f9d3c6ed9c03de55c0dd569462b7ec4b8d8555e556eed48e7ebd784d WatchSource:0}: Error finding container e08ba545f9d3c6ed9c03de55c0dd569462b7ec4b8d8555e556eed48e7ebd784d: Status 404 returned error can't find the container with id e08ba545f9d3c6ed9c03de55c0dd569462b7ec4b8d8555e556eed48e7ebd784d
Sep 11 16:59:07.909923 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.909831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" event={"ID":"ff81ba77-ad45-40ed-aa9e-4da8391b8492","Type":"ContainerStarted","Data":"4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca"}
Sep 11 16:59:07.909923 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:07.909916    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" event={"ID":"ff81ba77-ad45-40ed-aa9e-4da8391b8492","Type":"ContainerStarted","Data":"e08ba545f9d3c6ed9c03de55c0dd569462b7ec4b8d8555e556eed48e7ebd784d"}
Sep 11 16:59:09.540045 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.540003    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 11 16:59:09.783796 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.783761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:09.824754 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.824679    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"]
Sep 11 16:59:09.828059 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.828041    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:09.830792 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.830769    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 11 16:59:09.830929 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.830791    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 11 16:59:09.844471 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.844444    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"]
Sep 11 16:59:09.908453 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.908417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e48101-05db-47fb-bc48-713a979005c6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:09.908592 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.908511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5e48101-05db-47fb-bc48-713a979005c6-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:09.908592 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.908553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:09.908592 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.908586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqqsm\" (UniqueName: \"kubernetes.io/projected/a5e48101-05db-47fb-bc48-713a979005c6-kube-api-access-mqqsm\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:09.908746 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.908609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:09.917567 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.917544    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:59:09.918565 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.918539    2576 generic.go:356] "Generic (PLEG): container finished" podID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerID="fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e" exitCode=0
Sep 11 16:59:09.918669 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.918586    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" event={"ID":"372dbfef-4ffd-4dda-a47e-4a79dc274084","Type":"ContainerDied","Data":"fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e"}
Sep 11 16:59:09.918669 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.918631    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x" event={"ID":"372dbfef-4ffd-4dda-a47e-4a79dc274084","Type":"ContainerDied","Data":"6de5d05f4326d443e6aa5f34d81cc1d155a3198f55e64f60971757516eac3ac4"}
Sep 11 16:59:09.918669 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.918656    2576 scope.go:117] "RemoveContainer" containerID="3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20"
Sep 11 16:59:09.926925 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.926906    2576 scope.go:117] "RemoveContainer" containerID="fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e"
Sep 11 16:59:09.934372 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.934354    2576 scope.go:117] "RemoveContainer" containerID="2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666"
Sep 11 16:59:09.942633 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.942584    2576 scope.go:117] "RemoveContainer" containerID="3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20"
Sep 11 16:59:09.942977 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:59:09.942955    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20\": container with ID starting with 3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20 not found: ID does not exist" containerID="3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20"
Sep 11 16:59:09.943074 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.942992    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20"} err="failed to get container status \"3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20\": rpc error: code = NotFound desc = could not find container \"3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20\": container with ID starting with 3c9d5361848636f24a9deca1a0a685dfe4fe6f50c00b101ac6c7c508207fdc20 not found: ID does not exist"
Sep 11 16:59:09.943074 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.943019    2576 scope.go:117] "RemoveContainer" containerID="fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e"
Sep 11 16:59:09.943324 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:59:09.943303    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e\": container with ID starting with fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e not found: ID does not exist" containerID="fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e"
Sep 11 16:59:09.943421 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.943327    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e"} err="failed to get container status \"fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e\": rpc error: code = NotFound desc = could not find container \"fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e\": container with ID starting with fdf9c5ca72812355c9daa6509dc16f5305b4b7f87f5ceab608c0ab34775a963e not found: ID does not exist"
Sep 11 16:59:09.943421 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.943342    2576 scope.go:117] "RemoveContainer" containerID="2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666"
Sep 11 16:59:09.943655 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:59:09.943635    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666\": container with ID starting with 2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666 not found: ID does not exist" containerID="2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666"
Sep 11 16:59:09.943713 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:09.943663    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666"} err="failed to get container status \"2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666\": rpc error: code = NotFound desc = could not find container \"2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666\": container with ID starting with 2bb12829730d95339bdf56d534ede195e1ce07946f0bf7f30761c0a34744b666 not found: ID does not exist"
Sep 11 16:59:10.009334 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009300    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/372dbfef-4ffd-4dda-a47e-4a79dc274084-proxy-tls\") pod \"372dbfef-4ffd-4dda-a47e-4a79dc274084\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") "
Sep 11 16:59:10.009520 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009354    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwsnn\" (UniqueName: \"kubernetes.io/projected/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-api-access-mwsnn\") pod \"372dbfef-4ffd-4dda-a47e-4a79dc274084\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") "
Sep 11 16:59:10.009520 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009409    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/372dbfef-4ffd-4dda-a47e-4a79dc274084-kserve-provision-location\") pod \"372dbfef-4ffd-4dda-a47e-4a79dc274084\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") "
Sep 11 16:59:10.009520 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009442    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-cabundle-cert\") pod \"372dbfef-4ffd-4dda-a47e-4a79dc274084\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") "
Sep 11 16:59:10.009520 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009476    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-rbac-proxy-sar-config\") pod \"372dbfef-4ffd-4dda-a47e-4a79dc274084\" (UID: \"372dbfef-4ffd-4dda-a47e-4a79dc274084\") "
Sep 11 16:59:10.009743 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5e48101-05db-47fb-bc48-713a979005c6-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.009743 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.009743 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqqsm\" (UniqueName: \"kubernetes.io/projected/a5e48101-05db-47fb-bc48-713a979005c6-kube-api-access-mqqsm\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.009743 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.009743 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e48101-05db-47fb-bc48-713a979005c6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.010067 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009778    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372dbfef-4ffd-4dda-a47e-4a79dc274084-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "372dbfef-4ffd-4dda-a47e-4a79dc274084" (UID: "372dbfef-4ffd-4dda-a47e-4a79dc274084"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:59:10.010067 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009856    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "372dbfef-4ffd-4dda-a47e-4a79dc274084" (UID: "372dbfef-4ffd-4dda-a47e-4a79dc274084"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:59:10.010067 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.009870    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "372dbfef-4ffd-4dda-a47e-4a79dc274084" (UID: "372dbfef-4ffd-4dda-a47e-4a79dc274084"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:59:10.010067 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:59:10.009908    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 11 16:59:10.010067 ip-10-0-143-197 kubenswrapper[2576]: E0911 16:59:10.010011    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e48101-05db-47fb-bc48-713a979005c6-proxy-tls podName:a5e48101-05db-47fb-bc48-713a979005c6 nodeName:}" failed. No retries permitted until 2026-09-11 16:59:10.509990196 +0000 UTC m=+2544.419791342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a5e48101-05db-47fb-bc48-713a979005c6-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" (UID: "a5e48101-05db-47fb-bc48-713a979005c6") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 11 16:59:10.010371 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.010350    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.010500 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.010470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.010654 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.010635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5e48101-05db-47fb-bc48-713a979005c6-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.011710 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.011691    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-api-access-mwsnn" (OuterVolumeSpecName: "kube-api-access-mwsnn") pod "372dbfef-4ffd-4dda-a47e-4a79dc274084" (UID: "372dbfef-4ffd-4dda-a47e-4a79dc274084"). InnerVolumeSpecName "kube-api-access-mwsnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:59:10.011811 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.011711    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372dbfef-4ffd-4dda-a47e-4a79dc274084-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "372dbfef-4ffd-4dda-a47e-4a79dc274084" (UID: "372dbfef-4ffd-4dda-a47e-4a79dc274084"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:59:10.018646 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.018628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqqsm\" (UniqueName: \"kubernetes.io/projected/a5e48101-05db-47fb-bc48-713a979005c6-kube-api-access-mqqsm\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.110630 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.110538    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:59:10.110630 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.110568    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:59:10.110630 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.110579    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/372dbfef-4ffd-4dda-a47e-4a79dc274084-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:59:10.110630 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.110587    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwsnn\" (UniqueName: \"kubernetes.io/projected/372dbfef-4ffd-4dda-a47e-4a79dc274084-kube-api-access-mwsnn\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:59:10.110630 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.110596    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/372dbfef-4ffd-4dda-a47e-4a79dc274084-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 16:59:10.343568 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.343528    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:10.513581 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.513543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e48101-05db-47fb-bc48-713a979005c6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.516109 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.516092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e48101-05db-47fb-bc48-713a979005c6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.740731 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.740697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:10.923486 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.923458    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"
Sep 11 16:59:10.953828 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.953793    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"]
Sep 11 16:59:10.966716 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:10.966689    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x"]
Sep 11 16:59:11.025492 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:11.025199    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"]
Sep 11 16:59:11.027747 ip-10-0-143-197 kubenswrapper[2576]: W0911 16:59:11.027720    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e48101_05db_47fb_bc48_713a979005c6.slice/crio-ed3b7e91cff8c7f3d201b0bfcc8309dbf8b545bc0536d26051eed40ba020a2cf WatchSource:0}: Error finding container ed3b7e91cff8c7f3d201b0bfcc8309dbf8b545bc0536d26051eed40ba020a2cf: Status 404 returned error can't find the container with id ed3b7e91cff8c7f3d201b0bfcc8309dbf8b545bc0536d26051eed40ba020a2cf
Sep 11 16:59:11.929282 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:11.929188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" event={"ID":"a5e48101-05db-47fb-bc48-713a979005c6","Type":"ContainerStarted","Data":"ae34d39e2239315abca3377dfae18d0a75751b27f51194bbb725fa993abf29f2"}
Sep 11 16:59:11.929703 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:11.929306    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" event={"ID":"a5e48101-05db-47fb-bc48-713a979005c6","Type":"ContainerStarted","Data":"ed3b7e91cff8c7f3d201b0bfcc8309dbf8b545bc0536d26051eed40ba020a2cf"}
Sep 11 16:59:11.930792 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:11.930769    2576 generic.go:356] "Generic (PLEG): container finished" podID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerID="4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca" exitCode=0
Sep 11 16:59:11.930866 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:11.930818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" event={"ID":"ff81ba77-ad45-40ed-aa9e-4da8391b8492","Type":"ContainerDied","Data":"4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca"}
Sep 11 16:59:12.602750 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:12.602716    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" path="/var/lib/kubelet/pods/372dbfef-4ffd-4dda-a47e-4a79dc274084/volumes"
Sep 11 16:59:12.937266 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:12.937164    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" event={"ID":"ff81ba77-ad45-40ed-aa9e-4da8391b8492","Type":"ContainerStarted","Data":"45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23"}
Sep 11 16:59:12.937266 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:12.937236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" event={"ID":"ff81ba77-ad45-40ed-aa9e-4da8391b8492","Type":"ContainerStarted","Data":"3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473"}
Sep 11 16:59:12.937747 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:12.937641    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:12.937813 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:12.937796    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:12.939035 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:12.939012    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 11 16:59:12.960761 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:12.960715    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podStartSLOduration=2.006095576 podStartE2EDuration="5.960701486s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.95460591s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:59:12.959850795 +0000 UTC m=+2546.869651955" watchObservedRunningTime="2026-09-11 16:59:12.960701486 +0000 UTC m=+2546.870502647"
Sep 11 16:59:13.940833 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:13.940789    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 11 16:59:15.949703 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:15.949669    2576 generic.go:356] "Generic (PLEG): container finished" podID="a5e48101-05db-47fb-bc48-713a979005c6" containerID="ae34d39e2239315abca3377dfae18d0a75751b27f51194bbb725fa993abf29f2" exitCode=0
Sep 11 16:59:15.950121 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:15.949742    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" event={"ID":"a5e48101-05db-47fb-bc48-713a979005c6","Type":"ContainerDied","Data":"ae34d39e2239315abca3377dfae18d0a75751b27f51194bbb725fa993abf29f2"}
Sep 11 16:59:16.956200 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:16.956164    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" event={"ID":"a5e48101-05db-47fb-bc48-713a979005c6","Type":"ContainerStarted","Data":"1f6158389cf9146de31ac6eea1c45cb5ef7f38e14e930f4c1082fb2f93954e09"}
Sep 11 16:59:16.956584 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:16.956228    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" event={"ID":"a5e48101-05db-47fb-bc48-713a979005c6","Type":"ContainerStarted","Data":"1306decf3005b6a85dfc762d70d8e02ae1eccdd1f4ce90b45cda7de5d4f542ea"}
Sep 11 16:59:16.956584 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:16.956420    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:16.978501 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:16.978451    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podStartSLOduration=4.018618911 podStartE2EDuration="7.978436634s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959817723s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:59:16.977352243 +0000 UTC m=+2550.887153406" watchObservedRunningTime="2026-09-11 16:59:16.978436634 +0000 UTC m=+2550.888237796"
Sep 11 16:59:17.960007 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:17.959974    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:17.961436 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:17.961406    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 16:59:18.945753 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:18.945728    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 16:59:18.946312 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:18.946271    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 11 16:59:18.964306 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:18.964274    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 16:59:23.968840 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:23.968807    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 16:59:23.969344 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:23.969276    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 16:59:28.946387 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:28.946346    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 11 16:59:33.969799 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:33.969762    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 16:59:38.947129 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:38.947089    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 11 16:59:43.973094 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:43.973042    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 16:59:48.947064 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:48.947022    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 11 16:59:53.969253 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:53.969212    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 16:59:58.946505 ip-10-0-143-197 kubenswrapper[2576]: I0911 16:59:58.946465    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 11 17:00:03.970151 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:03.970110    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 17:00:08.947125 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:08.947043    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 11 17:00:13.969407 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:13.969366    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 17:00:18.947487 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:18.947458    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 17:00:23.970165 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:23.970113    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 17:00:27.069503 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:27.069471    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"]
Sep 11 17:00:27.069892 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:27.069699    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" containerID="cri-o://3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473" gracePeriod=30
Sep 11 17:00:27.069892 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:27.069734    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kube-rbac-proxy" containerID="cri-o://45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23" gracePeriod=30
Sep 11 17:00:27.086210 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:27.086178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:27.230345 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:27.230317    2576 generic.go:356] "Generic (PLEG): container finished" podID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerID="45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23" exitCode=2
Sep 11 17:00:27.230492 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:27.230368    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" event={"ID":"ff81ba77-ad45-40ed-aa9e-4da8391b8492","Type":"ContainerDied","Data":"45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23"}
Sep 11 17:00:28.942045 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:28.942003    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 11 17:00:28.946337 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:28.946314    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 11 17:00:29.598669 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:29.598614    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 11 17:00:30.704770 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.704748    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 17:00:30.837175 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.837103    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff81ba77-ad45-40ed-aa9e-4da8391b8492-proxy-tls\") pod \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") "
Sep 11 17:00:30.837175 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.837157    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kserve-provision-location\") pod \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") "
Sep 11 17:00:30.837401 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.837191    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-cabundle-cert\") pod \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") "
Sep 11 17:00:30.837401 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.837233    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8l6l\" (UniqueName: \"kubernetes.io/projected/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-api-access-n8l6l\") pod \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") "
Sep 11 17:00:30.837401 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.837272    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-rbac-proxy-sar-config\") pod \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\" (UID: \"ff81ba77-ad45-40ed-aa9e-4da8391b8492\") "
Sep 11 17:00:30.837573 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.837486    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff81ba77-ad45-40ed-aa9e-4da8391b8492" (UID: "ff81ba77-ad45-40ed-aa9e-4da8391b8492"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:30.837664 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.837643    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff81ba77-ad45-40ed-aa9e-4da8391b8492" (UID: "ff81ba77-ad45-40ed-aa9e-4da8391b8492"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:00:30.837716 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.837667    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff81ba77-ad45-40ed-aa9e-4da8391b8492" (UID: "ff81ba77-ad45-40ed-aa9e-4da8391b8492"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:00:30.839337 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.839319    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-api-access-n8l6l" (OuterVolumeSpecName: "kube-api-access-n8l6l") pod "ff81ba77-ad45-40ed-aa9e-4da8391b8492" (UID: "ff81ba77-ad45-40ed-aa9e-4da8391b8492"). InnerVolumeSpecName "kube-api-access-n8l6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:00:30.839449 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.839431    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff81ba77-ad45-40ed-aa9e-4da8391b8492-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff81ba77-ad45-40ed-aa9e-4da8391b8492" (UID: "ff81ba77-ad45-40ed-aa9e-4da8391b8492"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:00:30.937913 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.937886    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff81ba77-ad45-40ed-aa9e-4da8391b8492-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:30.937913 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.937908    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:30.938080 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.937919    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:30.938080 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.937927    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8l6l\" (UniqueName: \"kubernetes.io/projected/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-api-access-n8l6l\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:30.938080 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:30.937957    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff81ba77-ad45-40ed-aa9e-4da8391b8492-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:31.247620 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.247585    2576 generic.go:356] "Generic (PLEG): container finished" podID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerID="3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473" exitCode=0
Sep 11 17:00:31.247777 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.247635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" event={"ID":"ff81ba77-ad45-40ed-aa9e-4da8391b8492","Type":"ContainerDied","Data":"3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473"}
Sep 11 17:00:31.247777 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.247648    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"
Sep 11 17:00:31.247777 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.247697    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm" event={"ID":"ff81ba77-ad45-40ed-aa9e-4da8391b8492","Type":"ContainerDied","Data":"e08ba545f9d3c6ed9c03de55c0dd569462b7ec4b8d8555e556eed48e7ebd784d"}
Sep 11 17:00:31.247777 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.247726    2576 scope.go:117] "RemoveContainer" containerID="45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23"
Sep 11 17:00:31.256592 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.256573    2576 scope.go:117] "RemoveContainer" containerID="3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473"
Sep 11 17:00:31.263745 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.263730    2576 scope.go:117] "RemoveContainer" containerID="4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca"
Sep 11 17:00:31.270323 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.270304    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"]
Sep 11 17:00:31.271229 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.271216    2576 scope.go:117] "RemoveContainer" containerID="45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23"
Sep 11 17:00:31.271474 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:00:31.271456    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23\": container with ID starting with 45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23 not found: ID does not exist" containerID="45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23"
Sep 11 17:00:31.271522 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.271481    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23"} err="failed to get container status \"45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23\": rpc error: code = NotFound desc = could not find container \"45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23\": container with ID starting with 45a30bbdb5c4b23c936f0f931661ff92fd6f14f18b1d4c978ea9a229c7642a23 not found: ID does not exist"
Sep 11 17:00:31.271522 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.271496    2576 scope.go:117] "RemoveContainer" containerID="3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473"
Sep 11 17:00:31.271705 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:00:31.271690    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473\": container with ID starting with 3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473 not found: ID does not exist" containerID="3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473"
Sep 11 17:00:31.271746 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.271708    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473"} err="failed to get container status \"3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473\": rpc error: code = NotFound desc = could not find container \"3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473\": container with ID starting with 3c0c811dee09c896dfbf290eaa505f89eca196d34cab88dc39b95f6815a17473 not found: ID does not exist"
Sep 11 17:00:31.271746 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.271720    2576 scope.go:117] "RemoveContainer" containerID="4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca"
Sep 11 17:00:31.271926 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:00:31.271906    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca\": container with ID starting with 4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca not found: ID does not exist" containerID="4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca"
Sep 11 17:00:31.272006 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.271954    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca"} err="failed to get container status \"4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca\": rpc error: code = NotFound desc = could not find container \"4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca\": container with ID starting with 4125ce75a850208f261b36a30794a8660445fd114c9e103c0f760543d60339ca not found: ID does not exist"
Sep 11 17:00:31.275032 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:31.275013    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm"]
Sep 11 17:00:32.274104 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.274071    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:32.322439 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322403    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"]
Sep 11 17:00:32.322708 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322697    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="storage-initializer"
Sep 11 17:00:32.322754 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322710    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="storage-initializer"
Sep 11 17:00:32.322754 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322719    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="storage-initializer"
Sep 11 17:00:32.322754 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322725    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="storage-initializer"
Sep 11 17:00:32.322754 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322736    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kube-rbac-proxy"
Sep 11 17:00:32.322754 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322741    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kube-rbac-proxy"
Sep 11 17:00:32.322754 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322753    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container"
Sep 11 17:00:32.323007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322758    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container"
Sep 11 17:00:32.323007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322768    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container"
Sep 11 17:00:32.323007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322773    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container"
Sep 11 17:00:32.323007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322782    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kube-rbac-proxy"
Sep 11 17:00:32.323007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322787    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kube-rbac-proxy"
Sep 11 17:00:32.323007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322838    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kube-rbac-proxy"
Sep 11 17:00:32.323007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322847    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kube-rbac-proxy"
Sep 11 17:00:32.323007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322854    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="372dbfef-4ffd-4dda-a47e-4a79dc274084" containerName="kserve-container"
Sep 11 17:00:32.323007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.322860    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" containerName="kserve-container"
Sep 11 17:00:32.327345 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.327323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.330311 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.330281    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 11 17:00:32.330311 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.330285    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 11 17:00:32.330484 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.330343    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 11 17:00:32.339716 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.339250    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"]
Sep 11 17:00:32.449106 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.449073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.449287 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.449116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.449287 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.449232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.449287 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.449276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dblkn\" (UniqueName: \"kubernetes.io/projected/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-api-access-dblkn\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.550399 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.550318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.550399 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.550357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dblkn\" (UniqueName: \"kubernetes.io/projected/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-api-access-dblkn\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.550399 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.550382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.550669 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.550406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.550669 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:00:32.550515    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 11 17:00:32.550669 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:00:32.550586    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-proxy-tls podName:7e564d42-ab9f-4955-83a5-d8e6e76ca2c3 nodeName:}" failed. No retries permitted until 2026-09-11 17:00:33.050556487 +0000 UTC m=+2626.960357628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-proxy-tls") pod "isvc-sklearn-s3-predictor-84d46f45-8fxg6" (UID: "7e564d42-ab9f-4955-83a5-d8e6e76ca2c3") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 11 17:00:32.550834 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.550795    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.551094 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.551067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.560306 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.560287    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dblkn\" (UniqueName: \"kubernetes.io/projected/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-api-access-dblkn\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:32.602621 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.602590    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff81ba77-ad45-40ed-aa9e-4da8391b8492" path="/var/lib/kubelet/pods/ff81ba77-ad45-40ed-aa9e-4da8391b8492/volumes"
Sep 11 17:00:32.715382 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:32.715351    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:33.053610 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:33.053577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:33.056166 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:33.056148    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84d46f45-8fxg6\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:33.245048 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:33.245001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:33.520505 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:33.520472    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"]
Sep 11 17:00:33.523589 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:00:33.523565    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e564d42_ab9f_4955_83a5_d8e6e76ca2c3.slice/crio-acab69bb170a9618e8205bd65c75fa396884b986e668625c38b49d408811635d WatchSource:0}: Error finding container acab69bb170a9618e8205bd65c75fa396884b986e668625c38b49d408811635d: Status 404 returned error can't find the container with id acab69bb170a9618e8205bd65c75fa396884b986e668625c38b49d408811635d
Sep 11 17:00:34.262585 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:34.262550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" event={"ID":"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3","Type":"ContainerStarted","Data":"406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518"}
Sep 11 17:00:34.262794 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:34.262604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" event={"ID":"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3","Type":"ContainerStarted","Data":"acab69bb170a9618e8205bd65c75fa396884b986e668625c38b49d408811635d"}
Sep 11 17:00:35.268774 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:35.268735    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerID="406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518" exitCode=0
Sep 11 17:00:35.269220 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:35.268792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" event={"ID":"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3","Type":"ContainerDied","Data":"406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518"}
Sep 11 17:00:36.275341 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:36.275300    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" event={"ID":"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3","Type":"ContainerStarted","Data":"c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663"}
Sep 11 17:00:36.275788 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:36.275359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" event={"ID":"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3","Type":"ContainerStarted","Data":"76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103"}
Sep 11 17:00:36.275788 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:36.275436    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:36.298136 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:36.298089    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podStartSLOduration=3.194028799 podStartE2EDuration="4.298073818s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.104045019s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:00:36.295587899 +0000 UTC m=+2630.205389062" watchObservedRunningTime="2026-09-11 17:00:36.298073818 +0000 UTC m=+2630.207874982"
Sep 11 17:00:37.279052 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:37.279015    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:37.280334 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:37.280305    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:00:38.282676 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:38.282634    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:00:39.599128 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:39.599098    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 17:00:43.287213 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:43.287180    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:00:43.287847 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:43.287819    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:00:50.977407 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:50.977363    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"]
Sep 11 17:00:50.977929 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:50.977649    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container" containerID="cri-o://1306decf3005b6a85dfc762d70d8e02ae1eccdd1f4ce90b45cda7de5d4f542ea" gracePeriod=30
Sep 11 17:00:50.977929 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:50.977688    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kube-rbac-proxy" containerID="cri-o://1f6158389cf9146de31ac6eea1c45cb5ef7f38e14e930f4c1082fb2f93954e09" gracePeriod=30
Sep 11 17:00:50.977929 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:50.977873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:51.333111 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:51.333011    2576 generic.go:356] "Generic (PLEG): container finished" podID="a5e48101-05db-47fb-bc48-713a979005c6" containerID="1f6158389cf9146de31ac6eea1c45cb5ef7f38e14e930f4c1082fb2f93954e09" exitCode=2
Sep 11 17:00:51.333111 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:51.333072    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" event={"ID":"a5e48101-05db-47fb-bc48-713a979005c6","Type":"ContainerDied","Data":"1f6158389cf9146de31ac6eea1c45cb5ef7f38e14e930f4c1082fb2f93954e09"}
Sep 11 17:00:53.288105 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:53.288062    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:00:53.964616 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:53.964580    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Sep 11 17:00:56.301084 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.301028    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:56.354147 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.354064    2576 generic.go:356] "Generic (PLEG): container finished" podID="a5e48101-05db-47fb-bc48-713a979005c6" containerID="1306decf3005b6a85dfc762d70d8e02ae1eccdd1f4ce90b45cda7de5d4f542ea" exitCode=0
Sep 11 17:00:56.354147 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.354134    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" event={"ID":"a5e48101-05db-47fb-bc48-713a979005c6","Type":"ContainerDied","Data":"1306decf3005b6a85dfc762d70d8e02ae1eccdd1f4ce90b45cda7de5d4f542ea"}
Sep 11 17:00:56.355607 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.355581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:56.718475 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.718453    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 17:00:56.741053 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.741015    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqqsm\" (UniqueName: \"kubernetes.io/projected/a5e48101-05db-47fb-bc48-713a979005c6-kube-api-access-mqqsm\") pod \"a5e48101-05db-47fb-bc48-713a979005c6\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") "
Sep 11 17:00:56.741209 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.741063    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-kube-rbac-proxy-sar-config\") pod \"a5e48101-05db-47fb-bc48-713a979005c6\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") "
Sep 11 17:00:56.741209 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.741137    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-cabundle-cert\") pod \"a5e48101-05db-47fb-bc48-713a979005c6\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") "
Sep 11 17:00:56.741209 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.741171    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e48101-05db-47fb-bc48-713a979005c6-proxy-tls\") pod \"a5e48101-05db-47fb-bc48-713a979005c6\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") "
Sep 11 17:00:56.741414 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.741212    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5e48101-05db-47fb-bc48-713a979005c6-kserve-provision-location\") pod \"a5e48101-05db-47fb-bc48-713a979005c6\" (UID: \"a5e48101-05db-47fb-bc48-713a979005c6\") "
Sep 11 17:00:56.741789 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.741760    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e48101-05db-47fb-bc48-713a979005c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5e48101-05db-47fb-bc48-713a979005c6" (UID: "a5e48101-05db-47fb-bc48-713a979005c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:56.741882 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.741832    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5e48101-05db-47fb-bc48-713a979005c6" (UID: "a5e48101-05db-47fb-bc48-713a979005c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:00:56.742533 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.742507    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5e48101-05db-47fb-bc48-713a979005c6" (UID: "a5e48101-05db-47fb-bc48-713a979005c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:00:56.743680 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.743657    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e48101-05db-47fb-bc48-713a979005c6-kube-api-access-mqqsm" (OuterVolumeSpecName: "kube-api-access-mqqsm") pod "a5e48101-05db-47fb-bc48-713a979005c6" (UID: "a5e48101-05db-47fb-bc48-713a979005c6"). InnerVolumeSpecName "kube-api-access-mqqsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:00:56.744076 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.744054    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e48101-05db-47fb-bc48-713a979005c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5e48101-05db-47fb-bc48-713a979005c6" (UID: "a5e48101-05db-47fb-bc48-713a979005c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:00:56.842579 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.842549    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:56.842579 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.842578    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5e48101-05db-47fb-bc48-713a979005c6-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:56.842768 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.842592    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5e48101-05db-47fb-bc48-713a979005c6-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:56.842768 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.842603    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5e48101-05db-47fb-bc48-713a979005c6-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:56.842768 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:56.842614    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqqsm\" (UniqueName: \"kubernetes.io/projected/a5e48101-05db-47fb-bc48-713a979005c6-kube-api-access-mqqsm\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:00:57.359887 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:57.359856    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd" event={"ID":"a5e48101-05db-47fb-bc48-713a979005c6","Type":"ContainerDied","Data":"ed3b7e91cff8c7f3d201b0bfcc8309dbf8b545bc0536d26051eed40ba020a2cf"}
Sep 11 17:00:57.360356 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:57.359924    2576 scope.go:117] "RemoveContainer" containerID="1f6158389cf9146de31ac6eea1c45cb5ef7f38e14e930f4c1082fb2f93954e09"
Sep 11 17:00:57.360356 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:57.359867    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"
Sep 11 17:00:57.369008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:57.368990    2576 scope.go:117] "RemoveContainer" containerID="1306decf3005b6a85dfc762d70d8e02ae1eccdd1f4ce90b45cda7de5d4f542ea"
Sep 11 17:00:57.376145 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:57.376128    2576 scope.go:117] "RemoveContainer" containerID="ae34d39e2239315abca3377dfae18d0a75751b27f51194bbb725fa993abf29f2"
Sep 11 17:00:57.383459 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:57.383435    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"]
Sep 11 17:00:57.388043 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:57.388023    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd"]
Sep 11 17:00:58.602721 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:00:58.602683    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5e48101-05db-47fb-bc48-713a979005c6" path="/var/lib/kubelet/pods/a5e48101-05db-47fb-bc48-713a979005c6/volumes"
Sep 11 17:01:00.171555 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:00.171520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:03.288208 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:03.288171    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:01:13.288411 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:13.288366    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:01:23.288510 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:23.288462    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:01:33.287799 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:33.287757    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:01:43.288453 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:43.288362    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:01:46.197395 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:46.197365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:51.415379 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.415343    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:51.501459 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.501421    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"]
Sep 11 17:01:51.501716 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.501704    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kube-rbac-proxy"
Sep 11 17:01:51.501761 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.501718    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kube-rbac-proxy"
Sep 11 17:01:51.501761 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.501737    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container"
Sep 11 17:01:51.501761 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.501743    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container"
Sep 11 17:01:51.501761 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.501753    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="storage-initializer"
Sep 11 17:01:51.501761 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.501758    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="storage-initializer"
Sep 11 17:01:51.501924 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.501798    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kube-rbac-proxy"
Sep 11 17:01:51.501924 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.501807    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e48101-05db-47fb-bc48-713a979005c6" containerName="kserve-container"
Sep 11 17:01:51.506053 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.506033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.508816 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.508785    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 11 17:01:51.508960 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.508789    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 11 17:01:51.508960 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.508920    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:01:51.509109 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.508927    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 17:01:51.520331 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.520311    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"]
Sep 11 17:01:51.650254 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.650216    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.650440 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.650289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.650440 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.650344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlxsl\" (UniqueName: \"kubernetes.io/projected/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-api-access-nlxsl\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.650440 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.650378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.650440 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.650408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.751431 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.751402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.751590 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.751443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.751590 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.751497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.751590 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.751516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlxsl\" (UniqueName: \"kubernetes.io/projected/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-api-access-nlxsl\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.751590 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.751541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.751838 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:01:51.751609    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 11 17:01:51.751838 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:01:51.751691    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-proxy-tls podName:e0250562-17ca-4e21-9cd0-6266e8b0f5a8 nodeName:}" failed. No retries permitted until 2026-09-11 17:01:52.251670112 +0000 UTC m=+2706.161471271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" (UID: "e0250562-17ca-4e21-9cd0-6266e8b0f5a8") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 11 17:01:51.751838 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.751811    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.752286 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.752265    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.752330 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.752265    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.761419 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.761400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlxsl\" (UniqueName: \"kubernetes.io/projected/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-api-access-nlxsl\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:51.842776 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:51.842746    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:52.256644 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:52.256613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:52.259124 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:52.259097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:52.416453 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:52.416409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:52.696545 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:52.696513    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"]
Sep 11 17:01:52.700387 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:01:52.700359    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0250562_17ca_4e21_9cd0_6266e8b0f5a8.slice/crio-3095cd9b48549cf8cfcb5597f11167ae9736c37a0db63197d5810867f38f2c5d WatchSource:0}: Error finding container 3095cd9b48549cf8cfcb5597f11167ae9736c37a0db63197d5810867f38f2c5d: Status 404 returned error can't find the container with id 3095cd9b48549cf8cfcb5597f11167ae9736c37a0db63197d5810867f38f2c5d
Sep 11 17:01:53.288120 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:53.288091    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:01:53.579148 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:53.579056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" event={"ID":"e0250562-17ca-4e21-9cd0-6266e8b0f5a8","Type":"ContainerStarted","Data":"3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30"}
Sep 11 17:01:53.579148 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:53.579117    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" event={"ID":"e0250562-17ca-4e21-9cd0-6266e8b0f5a8","Type":"ContainerStarted","Data":"3095cd9b48549cf8cfcb5597f11167ae9736c37a0db63197d5810867f38f2c5d"}
Sep 11 17:01:56.591817 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:56.591782    2576 generic.go:356] "Generic (PLEG): container finished" podID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerID="3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30" exitCode=0
Sep 11 17:01:56.592270 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:56.591859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" event={"ID":"e0250562-17ca-4e21-9cd0-6266e8b0f5a8","Type":"ContainerDied","Data":"3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30"}
Sep 11 17:01:57.598300 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:57.598269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" event={"ID":"e0250562-17ca-4e21-9cd0-6266e8b0f5a8","Type":"ContainerStarted","Data":"952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304"}
Sep 11 17:01:57.598787 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:57.598335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" event={"ID":"e0250562-17ca-4e21-9cd0-6266e8b0f5a8","Type":"ContainerStarted","Data":"b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa"}
Sep 11 17:01:57.598787 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:57.598670    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:57.598908 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:57.598830    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:01:57.600072 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:57.600045    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 11 17:01:57.620229 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:57.620184    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" podStartSLOduration=2.992481369 podStartE2EDuration="6.620168894s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.627687525s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:01:57.618591637 +0000 UTC m=+2711.528392801" watchObservedRunningTime="2026-09-11 17:01:57.620168894 +0000 UTC m=+2711.529970055"
Sep 11 17:01:58.602495 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:01:58.602460    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 11 17:02:02.430520 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:02.430472    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"]
Sep 11 17:02:02.430900 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:02.430867    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" containerID="cri-o://76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103" gracePeriod=30
Sep 11 17:02:02.431002 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:02.430908    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kube-rbac-proxy" containerID="cri-o://c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663" gracePeriod=30
Sep 11 17:02:02.433225 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:02.433179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:02.567372 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:02.567333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:02.622606 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:02.622568    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerID="c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663" exitCode=2
Sep 11 17:02:02.622783 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:02.622619    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" event={"ID":"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3","Type":"ContainerDied","Data":"c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663"}
Sep 11 17:02:02.798953 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:02.798909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:03.283889 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:03.283838    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 11 17:02:03.288697 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:03.288670    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 11 17:02:03.607360 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:03.607272    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:02:03.608278 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:03.608248    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 11 17:02:06.980543 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:06.980518    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:02:07.169064 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.168973    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dblkn\" (UniqueName: \"kubernetes.io/projected/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-api-access-dblkn\") pod \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") "
Sep 11 17:02:07.169064 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.169037    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kserve-provision-location\") pod \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") "
Sep 11 17:02:07.169064 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.169058    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-proxy-tls\") pod \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") "
Sep 11 17:02:07.169366 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.169086    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-rbac-proxy-sar-config\") pod \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\" (UID: \"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3\") "
Sep 11 17:02:07.169366 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.169283    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" (UID: "7e564d42-ab9f-4955-83a5-d8e6e76ca2c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:02:07.169456 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.169412    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" (UID: "7e564d42-ab9f-4955-83a5-d8e6e76ca2c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:02:07.171264 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.171242    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" (UID: "7e564d42-ab9f-4955-83a5-d8e6e76ca2c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:02:07.171328 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.171262    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-api-access-dblkn" (OuterVolumeSpecName: "kube-api-access-dblkn") pod "7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" (UID: "7e564d42-ab9f-4955-83a5-d8e6e76ca2c3"). InnerVolumeSpecName "kube-api-access-dblkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:02:07.270556 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.270518    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dblkn\" (UniqueName: \"kubernetes.io/projected/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-api-access-dblkn\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:02:07.270556 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.270547    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:02:07.270556 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.270556    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:02:07.270556 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.270565    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:02:07.643221 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.643182    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerID="76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103" exitCode=0
Sep 11 17:02:07.643393 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.643231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" event={"ID":"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3","Type":"ContainerDied","Data":"76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103"}
Sep 11 17:02:07.643393 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.643243    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"
Sep 11 17:02:07.643393 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.643289    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6" event={"ID":"7e564d42-ab9f-4955-83a5-d8e6e76ca2c3","Type":"ContainerDied","Data":"acab69bb170a9618e8205bd65c75fa396884b986e668625c38b49d408811635d"}
Sep 11 17:02:07.643393 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.643326    2576 scope.go:117] "RemoveContainer" containerID="c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663"
Sep 11 17:02:07.651862 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.651843    2576 scope.go:117] "RemoveContainer" containerID="76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103"
Sep 11 17:02:07.659205 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.659186    2576 scope.go:117] "RemoveContainer" containerID="406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518"
Sep 11 17:02:07.666105 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.666090    2576 scope.go:117] "RemoveContainer" containerID="c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663"
Sep 11 17:02:07.666381 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:02:07.666363    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663\": container with ID starting with c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663 not found: ID does not exist" containerID="c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663"
Sep 11 17:02:07.666442 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.666389    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663"} err="failed to get container status \"c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663\": rpc error: code = NotFound desc = could not find container \"c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663\": container with ID starting with c38b925333a519249230f7a172388ee320cc02cbd064645b7a6cd1d7d3b8a663 not found: ID does not exist"
Sep 11 17:02:07.666442 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.666405    2576 scope.go:117] "RemoveContainer" containerID="76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103"
Sep 11 17:02:07.666634 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:02:07.666618    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103\": container with ID starting with 76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103 not found: ID does not exist" containerID="76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103"
Sep 11 17:02:07.666681 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.666636    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103"} err="failed to get container status \"76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103\": rpc error: code = NotFound desc = could not find container \"76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103\": container with ID starting with 76d7ca666880b80db9090e31794a07beb8e097c8740b49f94e3817ab7fd91103 not found: ID does not exist"
Sep 11 17:02:07.666681 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.666648    2576 scope.go:117] "RemoveContainer" containerID="406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518"
Sep 11 17:02:07.666883 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:02:07.666864    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518\": container with ID starting with 406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518 not found: ID does not exist" containerID="406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518"
Sep 11 17:02:07.666965 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.666893    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518"} err="failed to get container status \"406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518\": rpc error: code = NotFound desc = could not find container \"406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518\": container with ID starting with 406fa8940afbb816535186ff46bfe70b1e3f4acb5508913674c81ed47fef2518 not found: ID does not exist"
Sep 11 17:02:07.670127 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.670107    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"]
Sep 11 17:02:07.675387 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:07.675367    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6"]
Sep 11 17:02:08.602349 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:08.602309    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" path="/var/lib/kubelet/pods/7e564d42-ab9f-4955-83a5-d8e6e76ca2c3/volumes"
Sep 11 17:02:13.609746 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:13.609717    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:02:21.445724 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:21.445689    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"]
Sep 11 17:02:21.446218 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:21.445985    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kserve-container" containerID="cri-o://b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa" gracePeriod=30
Sep 11 17:02:21.446218 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:21.446099    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kube-rbac-proxy" containerID="cri-o://952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304" gracePeriod=30
Sep 11 17:02:21.476421 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:21.476390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:21.698529 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:21.698437    2576 generic.go:356] "Generic (PLEG): container finished" podID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerID="952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304" exitCode=2
Sep 11 17:02:21.698529 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:21.698497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" event={"ID":"e0250562-17ca-4e21-9cd0-6266e8b0f5a8","Type":"ContainerDied","Data":"952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304"}
Sep 11 17:02:22.090257 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.090231    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:02:22.177124 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.177094    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kserve-provision-location\") pod \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") "
Sep 11 17:02:22.177303 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.177141    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-proxy-tls\") pod \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") "
Sep 11 17:02:22.177303 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.177172    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlxsl\" (UniqueName: \"kubernetes.io/projected/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-api-access-nlxsl\") pod \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") "
Sep 11 17:02:22.177303 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.177199    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-cabundle-cert\") pod \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") "
Sep 11 17:02:22.177429 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.177334    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-rbac-proxy-sar-config\") pod \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\" (UID: \"e0250562-17ca-4e21-9cd0-6266e8b0f5a8\") "
Sep 11 17:02:22.177546 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.177520    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0250562-17ca-4e21-9cd0-6266e8b0f5a8" (UID: "e0250562-17ca-4e21-9cd0-6266e8b0f5a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:02:22.177614 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.177552    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0250562-17ca-4e21-9cd0-6266e8b0f5a8" (UID: "e0250562-17ca-4e21-9cd0-6266e8b0f5a8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:02:22.177669 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.177649    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0250562-17ca-4e21-9cd0-6266e8b0f5a8" (UID: "e0250562-17ca-4e21-9cd0-6266e8b0f5a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:02:22.179518 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.179495    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0250562-17ca-4e21-9cd0-6266e8b0f5a8" (UID: "e0250562-17ca-4e21-9cd0-6266e8b0f5a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:02:22.179610 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.179520    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-api-access-nlxsl" (OuterVolumeSpecName: "kube-api-access-nlxsl") pod "e0250562-17ca-4e21-9cd0-6266e8b0f5a8" (UID: "e0250562-17ca-4e21-9cd0-6266e8b0f5a8"). InnerVolumeSpecName "kube-api-access-nlxsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:02:22.278143 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.278058    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlxsl\" (UniqueName: \"kubernetes.io/projected/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-api-access-nlxsl\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:02:22.278143 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.278086    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:02:22.278143 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.278096    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:02:22.278143 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.278106    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:02:22.278143 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.278116    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0250562-17ca-4e21-9cd0-6266e8b0f5a8-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:02:22.704595 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.704556    2576 generic.go:356] "Generic (PLEG): container finished" podID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerID="b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa" exitCode=0
Sep 11 17:02:22.705017 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.704603    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" event={"ID":"e0250562-17ca-4e21-9cd0-6266e8b0f5a8","Type":"ContainerDied","Data":"b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa"}
Sep 11 17:02:22.705017 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.704666    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68" event={"ID":"e0250562-17ca-4e21-9cd0-6266e8b0f5a8","Type":"ContainerDied","Data":"3095cd9b48549cf8cfcb5597f11167ae9736c37a0db63197d5810867f38f2c5d"}
Sep 11 17:02:22.705017 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.704622    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"
Sep 11 17:02:22.705017 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.704693    2576 scope.go:117] "RemoveContainer" containerID="952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304"
Sep 11 17:02:22.712784 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.712767    2576 scope.go:117] "RemoveContainer" containerID="b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa"
Sep 11 17:02:22.720078 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.720055    2576 scope.go:117] "RemoveContainer" containerID="3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30"
Sep 11 17:02:22.727344 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.727324    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"]
Sep 11 17:02:22.727868 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.727852    2576 scope.go:117] "RemoveContainer" containerID="952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304"
Sep 11 17:02:22.728177 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:02:22.728157    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304\": container with ID starting with 952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304 not found: ID does not exist" containerID="952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304"
Sep 11 17:02:22.728232 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.728185    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304"} err="failed to get container status \"952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304\": rpc error: code = NotFound desc = could not find container \"952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304\": container with ID starting with 952399f3be15f9a93c0eb7c47452cf2bf5c1567d602e5fd29270064ba1b21304 not found: ID does not exist"
Sep 11 17:02:22.728232 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.728204    2576 scope.go:117] "RemoveContainer" containerID="b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa"
Sep 11 17:02:22.728444 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:02:22.728427    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa\": container with ID starting with b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa not found: ID does not exist" containerID="b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa"
Sep 11 17:02:22.728492 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.728449    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa"} err="failed to get container status \"b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa\": rpc error: code = NotFound desc = could not find container \"b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa\": container with ID starting with b21df6cce2af963952b7d3816620bc9dff8261324fff48a0f0653d56f58b70aa not found: ID does not exist"
Sep 11 17:02:22.728492 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.728464    2576 scope.go:117] "RemoveContainer" containerID="3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30"
Sep 11 17:02:22.728704 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:02:22.728687    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30\": container with ID starting with 3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30 not found: ID does not exist" containerID="3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30"
Sep 11 17:02:22.728756 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.728712    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30"} err="failed to get container status \"3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30\": rpc error: code = NotFound desc = could not find container \"3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30\": container with ID starting with 3e4d6657d6d12e6d1c6873bdd77a32edf913a475e3048251660c43f087618a30 not found: ID does not exist"
Sep 11 17:02:22.732251 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.732229    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68"]
Sep 11 17:02:22.754995 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:22.754965    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:24.602748 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:24.602716    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" path="/var/lib/kubelet/pods/e0250562-17ca-4e21-9cd0-6266e8b0f5a8/volumes"
Sep 11 17:02:26.594439 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:26.594402    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:26.878660 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:26.878576    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:31.856862 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:31.856827    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 17:02:31.857721 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:31.857700    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 17:02:31.863276 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:31.863259    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 17:02:31.863989 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:02:31.863970    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 17:03:16.709481 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:16.709395    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:21.919652 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.919620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:21.968611 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968578    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"]
Sep 11 17:03:21.968852 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968841    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="storage-initializer"
Sep 11 17:03:21.968898 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968854    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="storage-initializer"
Sep 11 17:03:21.968898 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968867    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="storage-initializer"
Sep 11 17:03:21.968898 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968872    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="storage-initializer"
Sep 11 17:03:21.968898 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968883    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kube-rbac-proxy"
Sep 11 17:03:21.968898 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968889    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kube-rbac-proxy"
Sep 11 17:03:21.968898 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968899    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kube-rbac-proxy"
Sep 11 17:03:21.969115 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968905    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kube-rbac-proxy"
Sep 11 17:03:21.969115 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968914    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kserve-container"
Sep 11 17:03:21.969115 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968918    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kserve-container"
Sep 11 17:03:21.969115 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968925    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container"
Sep 11 17:03:21.969115 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.968929    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container"
Sep 11 17:03:21.969115 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.969002    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kserve-container"
Sep 11 17:03:21.969115 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.969011    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kube-rbac-proxy"
Sep 11 17:03:21.969115 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.969017    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0250562-17ca-4e21-9cd0-6266e8b0f5a8" containerName="kube-rbac-proxy"
Sep 11 17:03:21.969115 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.969024    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e564d42-ab9f-4955-83a5-d8e6e76ca2c3" containerName="kserve-container"
Sep 11 17:03:21.972141 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.972111    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:21.975028 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.975006    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 17:03:21.975028 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.975024    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 17:03:21.975191 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.975006    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 11 17:03:21.975191 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.975053    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 17:03:21.975287 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.975241    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:03:21.975328 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.975314    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 11 17:03:21.975919 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.975902    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 17:03:21.986573 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:21.986543    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"]
Sep 11 17:03:22.075186 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.075152    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:22.118179 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.118148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.118330 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.118197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.118330 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.118254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38069194-4c5f-4f08-b8b9-906bdd6ff04e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.118330 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.118273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mf4\" (UniqueName: \"kubernetes.io/projected/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-api-access-86mf4\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.118451 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.118364    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.219126 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.219100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.219277 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.219139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.219277 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.219174    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38069194-4c5f-4f08-b8b9-906bdd6ff04e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.219277 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.219200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mf4\" (UniqueName: \"kubernetes.io/projected/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-api-access-86mf4\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.219277 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.219240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.219665 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.219644    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.219870 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.219846    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.219955 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.219886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.221704 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.221685    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38069194-4c5f-4f08-b8b9-906bdd6ff04e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.229269 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.229247    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mf4\" (UniqueName: \"kubernetes.io/projected/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-api-access-86mf4\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-nh268\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.281983 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.281955    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:22.555732 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.555707    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"]
Sep 11 17:03:22.558757 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:03:22.558730    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38069194_4c5f_4f08_b8b9_906bdd6ff04e.slice/crio-494f290efb66b6a40eda2d0fce13b00e1fe2fbfd2ce0f263a8c37be5b663f74f WatchSource:0}: Error finding container 494f290efb66b6a40eda2d0fce13b00e1fe2fbfd2ce0f263a8c37be5b663f74f: Status 404 returned error can't find the container with id 494f290efb66b6a40eda2d0fce13b00e1fe2fbfd2ce0f263a8c37be5b663f74f
Sep 11 17:03:22.642107 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.642074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:22.929584 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.929503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerStarted","Data":"3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d"}
Sep 11 17:03:22.929584 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:22.929571    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerStarted","Data":"494f290efb66b6a40eda2d0fce13b00e1fe2fbfd2ce0f263a8c37be5b663f74f"}
Sep 11 17:03:23.724424 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.724391    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"]
Sep 11 17:03:23.727825 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.727808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.730387 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.730347    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 11 17:03:23.730524 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.730450    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 11 17:03:23.730580 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.730529    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 11 17:03:23.740811 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.740790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"]
Sep 11 17:03:23.741187 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.741165    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:23.832959 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.832910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.833082 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.832993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.833082 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.833018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rndc\" (UniqueName: \"kubernetes.io/projected/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-api-access-5rndc\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.833082 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.833039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.933381 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.933353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.933765 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.933387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rndc\" (UniqueName: \"kubernetes.io/projected/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-api-access-5rndc\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.933765 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.933417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.933765 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.933481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.933765 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:03:23.933516    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 11 17:03:23.933765 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:03:23.933573    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-proxy-tls podName:ccc9fa90-e5ec-4712-a1dc-32114ef5055f nodeName:}" failed. No retries permitted until 2026-09-11 17:03:24.433554099 +0000 UTC m=+2798.343355239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw" (UID: "ccc9fa90-e5ec-4712-a1dc-32114ef5055f") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 11 17:03:23.933984 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.933854    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.934156 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.934137    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:23.944072 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:23.944049    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rndc\" (UniqueName: \"kubernetes.io/projected/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-api-access-5rndc\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:24.209054 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:24.209026    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:24.437300 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:24.437269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:24.439858 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:24.439834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:24.637902 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:24.637819    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:24.911340 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:24.911316    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"]
Sep 11 17:03:24.913643 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:03:24.913615    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc9fa90_e5ec_4712_a1dc_32114ef5055f.slice/crio-7a34cc45055817d999c96f5292d69f8330dd9fd520ade1d41b8ee93b1b78a3a2 WatchSource:0}: Error finding container 7a34cc45055817d999c96f5292d69f8330dd9fd520ade1d41b8ee93b1b78a3a2: Status 404 returned error can't find the container with id 7a34cc45055817d999c96f5292d69f8330dd9fd520ade1d41b8ee93b1b78a3a2
Sep 11 17:03:24.937561 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:24.937533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw" event={"ID":"ccc9fa90-e5ec-4712-a1dc-32114ef5055f","Type":"ContainerStarted","Data":"7a34cc45055817d999c96f5292d69f8330dd9fd520ade1d41b8ee93b1b78a3a2"}
Sep 11 17:03:25.942852 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:25.942815    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw" event={"ID":"ccc9fa90-e5ec-4712-a1dc-32114ef5055f","Type":"ContainerStarted","Data":"eb5a0fb7dae6bee99613911c8a64648db019e0ee90b85cef4cd8f0d317f174bc"}
Sep 11 17:03:26.948986 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:26.948930    2576 generic.go:356] "Generic (PLEG): container finished" podID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerID="3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d" exitCode=0
Sep 11 17:03:26.949362 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:26.949007    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerDied","Data":"3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d"}
Sep 11 17:03:27.012254 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:27.012234    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:03:27.957611 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:27.957557    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerStarted","Data":"c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d"}
Sep 11 17:03:28.964054 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:28.964017    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw_ccc9fa90-e5ec-4712-a1dc-32114ef5055f/storage-initializer/0.log"
Sep 11 17:03:28.964492 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:28.964074    2576 generic.go:356] "Generic (PLEG): container finished" podID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" containerID="eb5a0fb7dae6bee99613911c8a64648db019e0ee90b85cef4cd8f0d317f174bc" exitCode=1
Sep 11 17:03:28.964492 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:28.964153    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw" event={"ID":"ccc9fa90-e5ec-4712-a1dc-32114ef5055f","Type":"ContainerDied","Data":"eb5a0fb7dae6bee99613911c8a64648db019e0ee90b85cef4cd8f0d317f174bc"}
Sep 11 17:03:29.969892 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:29.969847    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerStarted","Data":"e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90"}
Sep 11 17:03:29.972190 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:29.972168    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw_ccc9fa90-e5ec-4712-a1dc-32114ef5055f/storage-initializer/0.log"
Sep 11 17:03:29.972302 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:29.972279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw" event={"ID":"ccc9fa90-e5ec-4712-a1dc-32114ef5055f","Type":"ContainerStarted","Data":"4481db4fef58eadcd27420206c6209475496fbbd5fbf8061f12c32b7e24550fb"}
Sep 11 17:03:30.978953 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:30.978905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerStarted","Data":"9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c"}
Sep 11 17:03:30.979380 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:30.979152    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:30.979380 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:30.979227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:30.979380 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:30.979263    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:31.005310 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:31.005267    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podStartSLOduration=3.264092597 podStartE2EDuration="10.005247417s" totalImagesPullingTime="2.882549657s" totalInitContainerRuntime="3.858605163s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:03:31.004794175 +0000 UTC m=+2804.914595341" watchObservedRunningTime="2026-09-11 17:03:31.005247417 +0000 UTC m=+2804.915048579"
Sep 11 17:03:33.724274 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:33.724239    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"]
Sep 11 17:03:33.724668 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:33.724442    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw" podUID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" containerName="storage-initializer" containerID="cri-o://4481db4fef58eadcd27420206c6209475496fbbd5fbf8061f12c32b7e24550fb" gracePeriod=30
Sep 11 17:03:33.738349 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:33.738325    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:34.946350 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:34.946317    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:35.118516 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:35.118484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:36.989793 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:36.989769    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:03:37.004189 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.004168    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw_ccc9fa90-e5ec-4712-a1dc-32114ef5055f/storage-initializer/1.log"
Sep 11 17:03:37.004620 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.004603    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw_ccc9fa90-e5ec-4712-a1dc-32114ef5055f/storage-initializer/0.log"
Sep 11 17:03:37.004691 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.004644    2576 generic.go:356] "Generic (PLEG): container finished" podID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" containerID="4481db4fef58eadcd27420206c6209475496fbbd5fbf8061f12c32b7e24550fb" exitCode=1
Sep 11 17:03:37.004753 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.004723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw" event={"ID":"ccc9fa90-e5ec-4712-a1dc-32114ef5055f","Type":"ContainerDied","Data":"4481db4fef58eadcd27420206c6209475496fbbd5fbf8061f12c32b7e24550fb"}
Sep 11 17:03:37.004817 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.004801    2576 scope.go:117] "RemoveContainer" containerID="eb5a0fb7dae6bee99613911c8a64648db019e0ee90b85cef4cd8f0d317f174bc"
Sep 11 17:03:37.061904 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.061886    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw_ccc9fa90-e5ec-4712-a1dc-32114ef5055f/storage-initializer/1.log"
Sep 11 17:03:37.064237 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.064221    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:37.139003 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.138973    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-rbac-proxy-sar-config\") pod \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") "
Sep 11 17:03:37.139182 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.139015    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rndc\" (UniqueName: \"kubernetes.io/projected/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-api-access-5rndc\") pod \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") "
Sep 11 17:03:37.139182 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.139049    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kserve-provision-location\") pod \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") "
Sep 11 17:03:37.139182 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.139078    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-proxy-tls\") pod \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\" (UID: \"ccc9fa90-e5ec-4712-a1dc-32114ef5055f\") "
Sep 11 17:03:37.139374 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.139334    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ccc9fa90-e5ec-4712-a1dc-32114ef5055f" (UID: "ccc9fa90-e5ec-4712-a1dc-32114ef5055f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:03:37.139431 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.139392    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ccc9fa90-e5ec-4712-a1dc-32114ef5055f" (UID: "ccc9fa90-e5ec-4712-a1dc-32114ef5055f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:03:37.141372 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.141347    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-api-access-5rndc" (OuterVolumeSpecName: "kube-api-access-5rndc") pod "ccc9fa90-e5ec-4712-a1dc-32114ef5055f" (UID: "ccc9fa90-e5ec-4712-a1dc-32114ef5055f"). InnerVolumeSpecName "kube-api-access-5rndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:03:37.141487 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.141400    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ccc9fa90-e5ec-4712-a1dc-32114ef5055f" (UID: "ccc9fa90-e5ec-4712-a1dc-32114ef5055f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:03:37.239576 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.239545    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:03:37.239576 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.239569    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:03:37.239576 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.239579    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rndc\" (UniqueName: \"kubernetes.io/projected/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kube-api-access-5rndc\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:03:37.239826 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:37.239588    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ccc9fa90-e5ec-4712-a1dc-32114ef5055f-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:03:38.010710 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:38.010683    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw_ccc9fa90-e5ec-4712-a1dc-32114ef5055f/storage-initializer/1.log"
Sep 11 17:03:38.011146 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:38.010733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw" event={"ID":"ccc9fa90-e5ec-4712-a1dc-32114ef5055f","Type":"ContainerDied","Data":"7a34cc45055817d999c96f5292d69f8330dd9fd520ade1d41b8ee93b1b78a3a2"}
Sep 11 17:03:38.011146 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:38.010786    2576 scope.go:117] "RemoveContainer" containerID="4481db4fef58eadcd27420206c6209475496fbbd5fbf8061f12c32b7e24550fb"
Sep 11 17:03:38.011146 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:38.010797    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"
Sep 11 17:03:38.051429 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:38.051400    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"]
Sep 11 17:03:38.056989 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:38.056968    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw"]
Sep 11 17:03:38.603199 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:03:38.603171    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" path="/var/lib/kubelet/pods/ccc9fa90-e5ec-4712-a1dc-32114ef5055f/volumes"
Sep 11 17:04:02.283272 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:02.283232    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 11 17:04:06.995023 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:06.994987    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:04:17.967496 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:17.967456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:36.996229 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:36.996200    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:04:42.023838 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:42.023792    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"]
Sep 11 17:04:42.024365 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:42.024158    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-container" containerID="cri-o://c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d" gracePeriod=30
Sep 11 17:04:42.024365 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:42.024185    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-agent" containerID="cri-o://e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90" gracePeriod=30
Sep 11 17:04:42.024365 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:42.024245    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy" containerID="cri-o://9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c" gracePeriod=30
Sep 11 17:04:42.043433 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:42.043404    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:42.258440 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:42.258410    2576 generic.go:356] "Generic (PLEG): container finished" podID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerID="9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c" exitCode=2
Sep 11 17:04:42.258616 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:42.258487    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerDied","Data":"9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c"}
Sep 11 17:04:44.268533 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:44.268503    2576 generic.go:356] "Generic (PLEG): container finished" podID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerID="c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d" exitCode=0
Sep 11 17:04:44.268837 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:44.268569    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerDied","Data":"c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d"}
Sep 11 17:04:46.984066 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:46.984021    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 11 17:04:46.991214 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:46.991183    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 11 17:04:47.142245 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.142207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:47.192891 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.192853    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"]
Sep 11 17:04:47.193178 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.193164    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" containerName="storage-initializer"
Sep 11 17:04:47.193178 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.193180    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" containerName="storage-initializer"
Sep 11 17:04:47.193270 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.193194    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" containerName="storage-initializer"
Sep 11 17:04:47.193270 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.193200    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" containerName="storage-initializer"
Sep 11 17:04:47.193270 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.193253    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" containerName="storage-initializer"
Sep 11 17:04:47.193270 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.193263    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccc9fa90-e5ec-4712-a1dc-32114ef5055f" containerName="storage-initializer"
Sep 11 17:04:47.196329 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.196310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.198993 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.198970    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 11 17:04:47.199104 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.199048    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 11 17:04:47.212520 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.212493    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"]
Sep 11 17:04:47.285081 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.284997    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:47.362343 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.362306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.362343 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.362345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfzwb\" (UniqueName: \"kubernetes.io/projected/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-api-access-nfzwb\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.362574 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.362369    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.362574 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.362448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.362574 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.362512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.463698 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.463660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.463853 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.463714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.463853 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.463827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfzwb\" (UniqueName: \"kubernetes.io/projected/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-api-access-nfzwb\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.463996 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.463871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.463996 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.463902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.464111 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.464099    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.464511 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.464489    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.464585 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.464553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.466501 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.466478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.474202 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.474177    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfzwb\" (UniqueName: \"kubernetes.io/projected/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-api-access-nfzwb\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xf4dn\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.507196 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.507167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:04:47.796913 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:47.796878    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"]
Sep 11 17:04:47.800167 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:04:47.800144    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22c4fac_8190_4bc7_aef1_0c99a0fdd491.slice/crio-f8b206f1bdaf1209c681fd6624132da4d27ab37b271525336fbd97da09101daf WatchSource:0}: Error finding container f8b206f1bdaf1209c681fd6624132da4d27ab37b271525336fbd97da09101daf: Status 404 returned error can't find the container with id f8b206f1bdaf1209c681fd6624132da4d27ab37b271525336fbd97da09101daf
Sep 11 17:04:48.285872 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:48.285837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" event={"ID":"c22c4fac-8190-4bc7-aef1-0c99a0fdd491","Type":"ContainerStarted","Data":"a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f"}
Sep 11 17:04:48.286303 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:48.285891    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" event={"ID":"c22c4fac-8190-4bc7-aef1-0c99a0fdd491","Type":"ContainerStarted","Data":"f8b206f1bdaf1209c681fd6624132da4d27ab37b271525336fbd97da09101daf"}
Sep 11 17:04:51.984089 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:51.984043    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 11 17:04:53.305772 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:53.305737    2576 generic.go:356] "Generic (PLEG): container finished" podID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerID="a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f" exitCode=0
Sep 11 17:04:53.306282 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:53.305816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" event={"ID":"c22c4fac-8190-4bc7-aef1-0c99a0fdd491","Type":"ContainerDied","Data":"a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f"}
Sep 11 17:04:56.983837 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:56.983793    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 11 17:04:56.984348 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:56.983951    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:04:56.990505 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:04:56.990474    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 11 17:05:01.984291 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:01.984241    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 11 17:05:05.187596 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:05.187563    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:06.053723 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:06.053684    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:06.370544 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:06.370461    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" event={"ID":"c22c4fac-8190-4bc7-aef1-0c99a0fdd491","Type":"ContainerStarted","Data":"abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2"}
Sep 11 17:05:06.370544 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:06.370520    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" event={"ID":"c22c4fac-8190-4bc7-aef1-0c99a0fdd491","Type":"ContainerStarted","Data":"a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543"}
Sep 11 17:05:06.371016 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:06.370718    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:05:06.396127 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:06.396063    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podStartSLOduration=2.5460366370000003 podStartE2EDuration="19.396046828s" totalImagesPullingTime="12.123679416s" totalInitContainerRuntime="4.726330775s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:05:06.394764536 +0000 UTC m=+2900.304565698" watchObservedRunningTime="2026-09-11 17:05:06.396046828 +0000 UTC m=+2900.305847990"
Sep 11 17:05:06.427011 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:06.426981    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:06.984271 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:06.984226    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 11 17:05:06.990738 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:06.990705    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 11 17:05:06.990859 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:06.990837    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:05:07.374894 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:07.374792    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:05:07.376016 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:07.375994    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 11 17:05:08.378674 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:08.378630    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 11 17:05:11.984296 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:11.984247    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 11 17:05:12.216705 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.216675    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:05:12.243818 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.243716    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86mf4\" (UniqueName: \"kubernetes.io/projected/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-api-access-86mf4\") pod \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") "
Sep 11 17:05:12.243818 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.243773    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-cabundle-cert\") pod \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") "
Sep 11 17:05:12.243818 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.243819    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-rbac-proxy-sar-config\") pod \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") "
Sep 11 17:05:12.244177 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.243862    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38069194-4c5f-4f08-b8b9-906bdd6ff04e-proxy-tls\") pod \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") "
Sep 11 17:05:12.244177 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.243911    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kserve-provision-location\") pod \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\" (UID: \"38069194-4c5f-4f08-b8b9-906bdd6ff04e\") "
Sep 11 17:05:12.244338 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.244234    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38069194-4c5f-4f08-b8b9-906bdd6ff04e" (UID: "38069194-4c5f-4f08-b8b9-906bdd6ff04e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:05:12.244398 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.244336    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38069194-4c5f-4f08-b8b9-906bdd6ff04e" (UID: "38069194-4c5f-4f08-b8b9-906bdd6ff04e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:05:12.244398 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.244338    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38069194-4c5f-4f08-b8b9-906bdd6ff04e" (UID: "38069194-4c5f-4f08-b8b9-906bdd6ff04e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:12.246563 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.246540    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-api-access-86mf4" (OuterVolumeSpecName: "kube-api-access-86mf4") pod "38069194-4c5f-4f08-b8b9-906bdd6ff04e" (UID: "38069194-4c5f-4f08-b8b9-906bdd6ff04e"). InnerVolumeSpecName "kube-api-access-86mf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:05:12.246674 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.246637    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38069194-4c5f-4f08-b8b9-906bdd6ff04e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38069194-4c5f-4f08-b8b9-906bdd6ff04e" (UID: "38069194-4c5f-4f08-b8b9-906bdd6ff04e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:05:12.344572 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.344534    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:05:12.344572 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.344562    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38069194-4c5f-4f08-b8b9-906bdd6ff04e-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:05:12.344572 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.344571    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:05:12.344572 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.344580    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86mf4\" (UniqueName: \"kubernetes.io/projected/38069194-4c5f-4f08-b8b9-906bdd6ff04e-kube-api-access-86mf4\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:05:12.344838 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.344589    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38069194-4c5f-4f08-b8b9-906bdd6ff04e-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:05:12.396585 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.396542    2576 generic.go:356] "Generic (PLEG): container finished" podID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerID="e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90" exitCode=137
Sep 11 17:05:12.396764 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.396617    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"
Sep 11 17:05:12.396764 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.396620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerDied","Data":"e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90"}
Sep 11 17:05:12.396764 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.396687    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268" event={"ID":"38069194-4c5f-4f08-b8b9-906bdd6ff04e","Type":"ContainerDied","Data":"494f290efb66b6a40eda2d0fce13b00e1fe2fbfd2ce0f263a8c37be5b663f74f"}
Sep 11 17:05:12.396764 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.396721    2576 scope.go:117] "RemoveContainer" containerID="9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c"
Sep 11 17:05:12.411655 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.406091    2576 scope.go:117] "RemoveContainer" containerID="e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90"
Sep 11 17:05:12.417745 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.417726    2576 scope.go:117] "RemoveContainer" containerID="c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d"
Sep 11 17:05:12.425494 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.425476    2576 scope.go:117] "RemoveContainer" containerID="3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d"
Sep 11 17:05:12.428666 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.428639    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"]
Sep 11 17:05:12.434036 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.434018    2576 scope.go:117] "RemoveContainer" containerID="9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c"
Sep 11 17:05:12.434350 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:05:12.434323    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c\": container with ID starting with 9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c not found: ID does not exist" containerID="9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c"
Sep 11 17:05:12.434404 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.434362    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c"} err="failed to get container status \"9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c\": rpc error: code = NotFound desc = could not find container \"9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c\": container with ID starting with 9e5af66d7729ff211ef790906a45ff75866d390e2d23737594ad17af6761e47c not found: ID does not exist"
Sep 11 17:05:12.434404 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.434381    2576 scope.go:117] "RemoveContainer" containerID="e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90"
Sep 11 17:05:12.434663 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:05:12.434648    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90\": container with ID starting with e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90 not found: ID does not exist" containerID="e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90"
Sep 11 17:05:12.434720 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.434666    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90"} err="failed to get container status \"e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90\": rpc error: code = NotFound desc = could not find container \"e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90\": container with ID starting with e0b48c2573e124fbe16b84ac2d6252e2262f7c40ab81aef63a836db169a0bb90 not found: ID does not exist"
Sep 11 17:05:12.434720 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.434678    2576 scope.go:117] "RemoveContainer" containerID="c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d"
Sep 11 17:05:12.434915 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:05:12.434901    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d\": container with ID starting with c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d not found: ID does not exist" containerID="c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d"
Sep 11 17:05:12.434989 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.434918    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d"} err="failed to get container status \"c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d\": rpc error: code = NotFound desc = could not find container \"c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d\": container with ID starting with c2b6d22df5aa7826812d9adeac81fc24027dca06899a1fe9d7c06633c9674c0d not found: ID does not exist"
Sep 11 17:05:12.434989 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.434929    2576 scope.go:117] "RemoveContainer" containerID="3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d"
Sep 11 17:05:12.435231 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:05:12.435209    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d\": container with ID starting with 3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d not found: ID does not exist" containerID="3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d"
Sep 11 17:05:12.435295 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.435239    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d"} err="failed to get container status \"3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d\": rpc error: code = NotFound desc = could not find container \"3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d\": container with ID starting with 3e9bbec221e4b75bbf70d69b78b4486a0c60c44b34a8b9c503d22dad9fde312d not found: ID does not exist"
Sep 11 17:05:12.435402 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.435384    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268"]
Sep 11 17:05:12.604275 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:12.604190    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" path="/var/lib/kubelet/pods/38069194-4c5f-4f08-b8b9-906bdd6ff04e/volumes"
Sep 11 17:05:13.383168 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:13.383139    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:05:13.383630 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:13.383607    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 11 17:05:16.033268 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:16.033236    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:17.210588 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.210543    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:17.235787 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.235758    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"]
Sep 11 17:05:17.236166 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236147    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="storage-initializer"
Sep 11 17:05:17.236261 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236168    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="storage-initializer"
Sep 11 17:05:17.236261 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236186    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-agent"
Sep 11 17:05:17.236261 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236195    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-agent"
Sep 11 17:05:17.236261 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236210    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-container"
Sep 11 17:05:17.236261 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236219    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-container"
Sep 11 17:05:17.236261 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236239    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy"
Sep 11 17:05:17.236261 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236249    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy"
Sep 11 17:05:17.236620 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236325    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-agent"
Sep 11 17:05:17.236620 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236352    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kube-rbac-proxy"
Sep 11 17:05:17.236620 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.236364    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="38069194-4c5f-4f08-b8b9-906bdd6ff04e" containerName="kserve-container"
Sep 11 17:05:17.241119 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.241092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.244074 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.244053    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 11 17:05:17.244200 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.244102    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 11 17:05:17.244200 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.244172    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 11 17:05:17.251313 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.251291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"]
Sep 11 17:05:17.280499 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.280465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.280499 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.280510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.280750 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.280552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.280750 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.280577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcjm\" (UniqueName: \"kubernetes.io/projected/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-api-access-bpcjm\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.280750 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.280607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.335213 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.335181    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:17.381249 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.381217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.381393 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.381263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.381393 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.381312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcjm\" (UniqueName: \"kubernetes.io/projected/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-api-access-bpcjm\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.381393 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.381333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.381393 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.381381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.382225 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.381802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.382333 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.382233    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.382333 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.382277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.384114 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.384097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.390555 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.390496    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcjm\" (UniqueName: \"kubernetes.io/projected/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-api-access-bpcjm\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.553625 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.553542    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:17.829057 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:17.829011    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"]
Sep 11 17:05:17.831705 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:05:17.831674    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0d6c57_19f7_4fd8_8f29_bf802d8a6972.slice/crio-f0b2856bbff78e1228b4f9c47fe2e416f3ffb00fdf181fdf543ce4f952694173 WatchSource:0}: Error finding container f0b2856bbff78e1228b4f9c47fe2e416f3ffb00fdf181fdf543ce4f952694173: Status 404 returned error can't find the container with id f0b2856bbff78e1228b4f9c47fe2e416f3ffb00fdf181fdf543ce4f952694173
Sep 11 17:05:18.423869 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:18.423821    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" event={"ID":"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972","Type":"ContainerStarted","Data":"c49d9d75e18e237ce62ab6f3cd6602cda27364aeed7aaac954d11e3ac036cc91"}
Sep 11 17:05:18.424304 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:18.423909    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" event={"ID":"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972","Type":"ContainerStarted","Data":"f0b2856bbff78e1228b4f9c47fe2e416f3ffb00fdf181fdf543ce4f952694173"}
Sep 11 17:05:19.429485 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:19.429453    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerID="c49d9d75e18e237ce62ab6f3cd6602cda27364aeed7aaac954d11e3ac036cc91" exitCode=0
Sep 11 17:05:19.429909 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:19.429502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" event={"ID":"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972","Type":"ContainerDied","Data":"c49d9d75e18e237ce62ab6f3cd6602cda27364aeed7aaac954d11e3ac036cc91"}
Sep 11 17:05:20.435072 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:20.435025    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" event={"ID":"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972","Type":"ContainerStarted","Data":"2794f4ff4514405be304f3fcec5817557c2a099ab4ae74ff84cdf592bb3fd0f1"}
Sep 11 17:05:20.435473 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:20.435136    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" event={"ID":"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972","Type":"ContainerStarted","Data":"393310e8358ea891533253afdfaa8a4814aa9e5c8dd37bbb213157d2e24d8cf1"}
Sep 11 17:05:20.435473 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:20.435285    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:20.435473 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:20.435428    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:20.436770 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:20.436742    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:05:20.459151 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:20.459107    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podStartSLOduration=2.4561521490000002 podStartE2EDuration="3.459093369s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.00294122s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:05:20.457403807 +0000 UTC m=+2914.367204995" watchObservedRunningTime="2026-09-11 17:05:20.459093369 +0000 UTC m=+2914.368894530"
Sep 11 17:05:21.438947 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:21.438897    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:05:23.383895 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:23.383855    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 11 17:05:26.443695 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:26.443666    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:05:26.444131 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:26.444106    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:05:33.383795 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:33.383744    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 11 17:05:36.444730 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:36.444694    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:05:43.383622 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:43.383585    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 11 17:05:46.444438 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:46.444398    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:05:53.385109 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:53.385073    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:05:56.444879 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:56.444845    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:05:58.257074 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:58.257042    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"]
Sep 11 17:05:58.257475 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:58.257242    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container" containerID="cri-o://a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543" gracePeriod=30
Sep 11 17:05:58.257475 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:58.257334    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kube-rbac-proxy" containerID="cri-o://abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2" gracePeriod=30
Sep 11 17:05:58.282571 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:58.282484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:58.379083 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:58.379047    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Sep 11 17:05:58.586226 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:58.586139    2576 generic.go:356] "Generic (PLEG): container finished" podID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerID="abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2" exitCode=2
Sep 11 17:05:58.586226 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:05:58.586177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" event={"ID":"c22c4fac-8190-4bc7-aef1-0c99a0fdd491","Type":"ContainerDied","Data":"abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2"}
Sep 11 17:06:00.993237 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:00.993208    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:06:01.129530 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.129448    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kserve-provision-location\") pod \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") "
Sep 11 17:06:01.129530 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.129495    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-cabundle-cert\") pod \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") "
Sep 11 17:06:01.129530 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.129529    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-proxy-tls\") pod \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") "
Sep 11 17:06:01.129802 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.129545    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfzwb\" (UniqueName: \"kubernetes.io/projected/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-api-access-nfzwb\") pod \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") "
Sep 11 17:06:01.129802 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.129574    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-rbac-proxy-sar-config\") pod \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\" (UID: \"c22c4fac-8190-4bc7-aef1-0c99a0fdd491\") "
Sep 11 17:06:01.129994 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.129963    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c22c4fac-8190-4bc7-aef1-0c99a0fdd491" (UID: "c22c4fac-8190-4bc7-aef1-0c99a0fdd491"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:06:01.130082 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.130038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c22c4fac-8190-4bc7-aef1-0c99a0fdd491" (UID: "c22c4fac-8190-4bc7-aef1-0c99a0fdd491"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:06:01.131782 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.131759    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c22c4fac-8190-4bc7-aef1-0c99a0fdd491" (UID: "c22c4fac-8190-4bc7-aef1-0c99a0fdd491"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:06:01.131891 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.131803    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-api-access-nfzwb" (OuterVolumeSpecName: "kube-api-access-nfzwb") pod "c22c4fac-8190-4bc7-aef1-0c99a0fdd491" (UID: "c22c4fac-8190-4bc7-aef1-0c99a0fdd491"). InnerVolumeSpecName "kube-api-access-nfzwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:06:01.139007 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.138981    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c22c4fac-8190-4bc7-aef1-0c99a0fdd491" (UID: "c22c4fac-8190-4bc7-aef1-0c99a0fdd491"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:01.230964 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.230923    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:01.230964 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.230964    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:01.231146 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.230975    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:01.231146 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.230982    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfzwb\" (UniqueName: \"kubernetes.io/projected/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-api-access-nfzwb\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:01.231146 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.230991    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22c4fac-8190-4bc7-aef1-0c99a0fdd491-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:01.600603 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.600564    2576 generic.go:356] "Generic (PLEG): container finished" podID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerID="a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543" exitCode=0
Sep 11 17:06:01.600766 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.600615    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"
Sep 11 17:06:01.600766 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.600652    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" event={"ID":"c22c4fac-8190-4bc7-aef1-0c99a0fdd491","Type":"ContainerDied","Data":"a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543"}
Sep 11 17:06:01.600766 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.600696    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn" event={"ID":"c22c4fac-8190-4bc7-aef1-0c99a0fdd491","Type":"ContainerDied","Data":"f8b206f1bdaf1209c681fd6624132da4d27ab37b271525336fbd97da09101daf"}
Sep 11 17:06:01.600766 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.600720    2576 scope.go:117] "RemoveContainer" containerID="abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2"
Sep 11 17:06:01.609461 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.609446    2576 scope.go:117] "RemoveContainer" containerID="a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543"
Sep 11 17:06:01.616912 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.616894    2576 scope.go:117] "RemoveContainer" containerID="a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f"
Sep 11 17:06:01.625462 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.625439    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"]
Sep 11 17:06:01.627990 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.627975    2576 scope.go:117] "RemoveContainer" containerID="abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2"
Sep 11 17:06:01.628294 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:06:01.628270    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2\": container with ID starting with abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2 not found: ID does not exist" containerID="abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2"
Sep 11 17:06:01.628408 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.628303    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2"} err="failed to get container status \"abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2\": rpc error: code = NotFound desc = could not find container \"abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2\": container with ID starting with abb765e6e278441f263542182a0dff1fb06f1e1a0d9a3555b26e46496df061c2 not found: ID does not exist"
Sep 11 17:06:01.628408 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.628326    2576 scope.go:117] "RemoveContainer" containerID="a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543"
Sep 11 17:06:01.628609 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:06:01.628589    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543\": container with ID starting with a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543 not found: ID does not exist" containerID="a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543"
Sep 11 17:06:01.628650 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.628617    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543"} err="failed to get container status \"a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543\": rpc error: code = NotFound desc = could not find container \"a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543\": container with ID starting with a44769981488cac68087854f5882e8d9ebf0587c3e6345aa84e8eed833c5c543 not found: ID does not exist"
Sep 11 17:06:01.628650 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.628638    2576 scope.go:117] "RemoveContainer" containerID="a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f"
Sep 11 17:06:01.628912 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:06:01.628894    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f\": container with ID starting with a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f not found: ID does not exist" containerID="a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f"
Sep 11 17:06:01.629019 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.628917    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f"} err="failed to get container status \"a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f\": rpc error: code = NotFound desc = could not find container \"a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f\": container with ID starting with a915d7b6bcb9e47c61a9e3c92c79d90247a6d91fb221f08674e8c49fe995536f not found: ID does not exist"
Sep 11 17:06:01.638960 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:01.638922    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn"]
Sep 11 17:06:02.603231 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:02.602737    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" path="/var/lib/kubelet/pods/c22c4fac-8190-4bc7-aef1-0c99a0fdd491/volumes"
Sep 11 17:06:03.428314 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.428283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:03.472719 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.472687    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"]
Sep 11 17:06:03.472996 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.472983    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kube-rbac-proxy"
Sep 11 17:06:03.472996 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.472997    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kube-rbac-proxy"
Sep 11 17:06:03.473110 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.473019    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="storage-initializer"
Sep 11 17:06:03.473110 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.473024    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="storage-initializer"
Sep 11 17:06:03.473110 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.473034    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container"
Sep 11 17:06:03.473110 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.473039    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container"
Sep 11 17:06:03.473110 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.473095    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kube-rbac-proxy"
Sep 11 17:06:03.473110 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.473106    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22c4fac-8190-4bc7-aef1-0c99a0fdd491" containerName="kserve-container"
Sep 11 17:06:03.476282 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.476265    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.479029 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.479003    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 17:06:03.479170 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.479073    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 11 17:06:03.479170 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.479141    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 11 17:06:03.489593 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.489567    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"]
Sep 11 17:06:03.549630 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.549594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2dzb\" (UniqueName: \"kubernetes.io/projected/e6813771-3140-47aa-b4b4-26b73be882d3-kube-api-access-w2dzb\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.549808 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.549642    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.549808 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.549665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.549808 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.549740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6813771-3140-47aa-b4b4-26b73be882d3-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.549808 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.549781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6813771-3140-47aa-b4b4-26b73be882d3-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.645981 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.645921    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:03.651072 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.651039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2dzb\" (UniqueName: \"kubernetes.io/projected/e6813771-3140-47aa-b4b4-26b73be882d3-kube-api-access-w2dzb\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.651215 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.651093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.651294 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.651226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.651294 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.651287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6813771-3140-47aa-b4b4-26b73be882d3-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.651404 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.651329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6813771-3140-47aa-b4b4-26b73be882d3-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.651658 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.651639    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6813771-3140-47aa-b4b4-26b73be882d3-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.651800 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.651782    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.651901 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.651882    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.653920 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.653901    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6813771-3140-47aa-b4b4-26b73be882d3-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.661427 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.661409    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2dzb\" (UniqueName: \"kubernetes.io/projected/e6813771-3140-47aa-b4b4-26b73be882d3-kube-api-access-w2dzb\") pod \"isvc-paddle-runtime-predictor-58b99fff76-njqm2\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:03.785973 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:03.785913    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:04.058023 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:04.057997    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"]
Sep 11 17:06:04.060468 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:06:04.060440    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6813771_3140_47aa_b4b4_26b73be882d3.slice/crio-e66ad975cb21e860f7e47b93200f436bf0770472493b1c550107e69d5f2cea91 WatchSource:0}: Error finding container e66ad975cb21e860f7e47b93200f436bf0770472493b1c550107e69d5f2cea91: Status 404 returned error can't find the container with id e66ad975cb21e860f7e47b93200f436bf0770472493b1c550107e69d5f2cea91
Sep 11 17:06:04.616206 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:04.616171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" event={"ID":"e6813771-3140-47aa-b4b4-26b73be882d3","Type":"ContainerStarted","Data":"eae42b28ebe0cf33de5c5ccf4fcf10d955f65d7b9d19a98c2dee3935302fd00a"}
Sep 11 17:06:04.616403 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:04.616230    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" event={"ID":"e6813771-3140-47aa-b4b4-26b73be882d3","Type":"ContainerStarted","Data":"e66ad975cb21e860f7e47b93200f436bf0770472493b1c550107e69d5f2cea91"}
Sep 11 17:06:06.444750 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:06.444713    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:06:09.636952 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:09.636910    2576 generic.go:356] "Generic (PLEG): container finished" podID="e6813771-3140-47aa-b4b4-26b73be882d3" containerID="eae42b28ebe0cf33de5c5ccf4fcf10d955f65d7b9d19a98c2dee3935302fd00a" exitCode=0
Sep 11 17:06:09.639368 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:09.636984    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" event={"ID":"e6813771-3140-47aa-b4b4-26b73be882d3","Type":"ContainerDied","Data":"eae42b28ebe0cf33de5c5ccf4fcf10d955f65d7b9d19a98c2dee3935302fd00a"}
Sep 11 17:06:10.642380 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:10.642341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" event={"ID":"e6813771-3140-47aa-b4b4-26b73be882d3","Type":"ContainerStarted","Data":"e2e5276dfa11b040185d45f1cdf4fc9b9b118f7a13c1b432f0db22979978d027"}
Sep 11 17:06:10.642864 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:10.642415    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" event={"ID":"e6813771-3140-47aa-b4b4-26b73be882d3","Type":"ContainerStarted","Data":"74a8df534e65ccb256059ba504b883017955a9694af8adcbc8ace25cd9c0e13f"}
Sep 11 17:06:10.642864 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:10.642639    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:10.663840 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:10.663793    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podStartSLOduration=2.807474393 podStartE2EDuration="7.663779722s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.856305329s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:06:10.663725009 +0000 UTC m=+2964.573526170" watchObservedRunningTime="2026-09-11 17:06:10.663779722 +0000 UTC m=+2964.573580949"
Sep 11 17:06:11.646665 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:11.646634    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:11.647924 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:11.647893    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 11 17:06:12.650505 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:12.650466    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 11 17:06:16.444772 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:16.444732    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:06:17.655067 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:17.655031    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:17.655612 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:17.655585    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 11 17:06:26.445100 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:26.445056    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:06:27.656009 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:27.655971    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 11 17:06:31.599034 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:31.598989    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:06:37.655776 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:37.655734    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 11 17:06:41.600121 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:41.600090    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:06:47.240681 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:47.240646    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"]
Sep 11 17:06:47.241238 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:47.240874    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" containerID="cri-o://393310e8358ea891533253afdfaa8a4814aa9e5c8dd37bbb213157d2e24d8cf1" gracePeriod=30
Sep 11 17:06:47.241238 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:47.240926    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kube-rbac-proxy" containerID="cri-o://2794f4ff4514405be304f3fcec5817557c2a099ab4ae74ff84cdf592bb3fd0f1" gracePeriod=30
Sep 11 17:06:47.248900 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:47.248876    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:47.656475 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:47.656385    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 11 17:06:47.791361 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:47.791324    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerID="2794f4ff4514405be304f3fcec5817557c2a099ab4ae74ff84cdf592bb3fd0f1" exitCode=2
Sep 11 17:06:47.791533 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:47.791398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" event={"ID":"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972","Type":"ContainerDied","Data":"2794f4ff4514405be304f3fcec5817557c2a099ab4ae74ff84cdf592bb3fd0f1"}
Sep 11 17:06:48.279070 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:48.279029    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:48.750000 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:48.749960    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:51.439951 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.439898    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 11 17:06:51.598910 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.598871    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 11 17:06:51.811914 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.811882    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerID="393310e8358ea891533253afdfaa8a4814aa9e5c8dd37bbb213157d2e24d8cf1" exitCode=0
Sep 11 17:06:51.812052 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.811963    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" event={"ID":"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972","Type":"ContainerDied","Data":"393310e8358ea891533253afdfaa8a4814aa9e5c8dd37bbb213157d2e24d8cf1"}
Sep 11 17:06:51.884019 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.883996    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:06:51.924826 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.924804    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpcjm\" (UniqueName: \"kubernetes.io/projected/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-api-access-bpcjm\") pod \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") "
Sep 11 17:06:51.924969 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.924842    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kserve-provision-location\") pod \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") "
Sep 11 17:06:51.924969 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.924868    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-proxy-tls\") pod \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") "
Sep 11 17:06:51.924969 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.924918    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-rbac-proxy-sar-config\") pod \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") "
Sep 11 17:06:51.924969 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.924959    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-cabundle-cert\") pod \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\" (UID: \"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972\") "
Sep 11 17:06:51.925209 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.925164    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" (UID: "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:51.925309 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.925287    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" (UID: "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:06:51.925394 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.925369    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" (UID: "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:06:51.927157 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.927136    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" (UID: "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:06:51.927247 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:51.927159    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-api-access-bpcjm" (OuterVolumeSpecName: "kube-api-access-bpcjm") pod "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" (UID: "2f0d6c57-19f7-4fd8-8f29-bf802d8a6972"). InnerVolumeSpecName "kube-api-access-bpcjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:06:52.026441 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.026378    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:52.026441 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.026403    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:52.026441 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.026413    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpcjm\" (UniqueName: \"kubernetes.io/projected/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kube-api-access-bpcjm\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:52.026441 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.026421    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:52.026441 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.026430    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:06:52.818593 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.818569    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"
Sep 11 17:06:52.819008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.818568    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8" event={"ID":"2f0d6c57-19f7-4fd8-8f29-bf802d8a6972","Type":"ContainerDied","Data":"f0b2856bbff78e1228b4f9c47fe2e416f3ffb00fdf181fdf543ce4f952694173"}
Sep 11 17:06:52.819008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.818702    2576 scope.go:117] "RemoveContainer" containerID="2794f4ff4514405be304f3fcec5817557c2a099ab4ae74ff84cdf592bb3fd0f1"
Sep 11 17:06:52.826864 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.826847    2576 scope.go:117] "RemoveContainer" containerID="393310e8358ea891533253afdfaa8a4814aa9e5c8dd37bbb213157d2e24d8cf1"
Sep 11 17:06:52.834243 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.834227    2576 scope.go:117] "RemoveContainer" containerID="c49d9d75e18e237ce62ab6f3cd6602cda27364aeed7aaac954d11e3ac036cc91"
Sep 11 17:06:52.839699 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.839678    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"]
Sep 11 17:06:52.844173 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:52.844150    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8"]
Sep 11 17:06:54.607541 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:54.607493    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" path="/var/lib/kubelet/pods/2f0d6c57-19f7-4fd8-8f29-bf802d8a6972/volumes"
Sep 11 17:06:57.656091 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:57.656061    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:06:58.321519 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:58.321483    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:59.420567 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.420530    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:59.454081 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.454047    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"]
Sep 11 17:06:59.454354 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.454341    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container"
Sep 11 17:06:59.454354 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.454355    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container"
Sep 11 17:06:59.454443 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.454372    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kube-rbac-proxy"
Sep 11 17:06:59.454443 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.454378    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kube-rbac-proxy"
Sep 11 17:06:59.454443 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.454395    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="storage-initializer"
Sep 11 17:06:59.454443 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.454400    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="storage-initializer"
Sep 11 17:06:59.454584 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.454457    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kube-rbac-proxy"
Sep 11 17:06:59.454584 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.454466    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f0d6c57-19f7-4fd8-8f29-bf802d8a6972" containerName="kserve-container"
Sep 11 17:06:59.457535 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.457515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.460215 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.460190    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 11 17:06:59.460327 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.460199    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 11 17:06:59.468439 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.468414    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"]
Sep 11 17:06:59.580865 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.580826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.581101 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.580888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12700993-9eef-47bb-aa76-fb84a21a3d3f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.581101 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.580930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.581101 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.581021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12700993-9eef-47bb-aa76-fb84a21a3d3f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.581101 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.581085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29p87\" (UniqueName: \"kubernetes.io/projected/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-api-access-29p87\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.682488 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.682388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29p87\" (UniqueName: \"kubernetes.io/projected/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-api-access-29p87\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.682488 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.682442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.682800 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.682530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12700993-9eef-47bb-aa76-fb84a21a3d3f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.682800 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.682559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.682800 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.682584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12700993-9eef-47bb-aa76-fb84a21a3d3f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.683099 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.683073    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12700993-9eef-47bb-aa76-fb84a21a3d3f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.683389 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.683365    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.683453 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.683366    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.685537 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.685515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12700993-9eef-47bb-aa76-fb84a21a3d3f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.698217 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.698193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29p87\" (UniqueName: \"kubernetes.io/projected/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-api-access-29p87\") pod \"isvc-pmml-runtime-predictor-885fc8777-lfvrb\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:06:59.765014 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.764973    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:59.768022 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:06:59.768002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:07:00.041358 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:00.041331    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"]
Sep 11 17:07:00.043968 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:07:00.043929    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12700993_9eef_47bb_aa76_fb84a21a3d3f.slice/crio-5418d8d415c60916da5bfc3926c8e1d2af43c6cc915371496685929df586652e WatchSource:0}: Error finding container 5418d8d415c60916da5bfc3926c8e1d2af43c6cc915371496685929df586652e: Status 404 returned error can't find the container with id 5418d8d415c60916da5bfc3926c8e1d2af43c6cc915371496685929df586652e
Sep 11 17:07:00.849811 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:00.849764    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" event={"ID":"12700993-9eef-47bb-aa76-fb84a21a3d3f","Type":"ContainerStarted","Data":"18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d"}
Sep 11 17:07:00.850311 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:00.849847    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" event={"ID":"12700993-9eef-47bb-aa76-fb84a21a3d3f","Type":"ContainerStarted","Data":"5418d8d415c60916da5bfc3926c8e1d2af43c6cc915371496685929df586652e"}
Sep 11 17:07:04.627414 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:04.627380    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"]
Sep 11 17:07:04.627865 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:04.627612    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container" containerID="cri-o://74a8df534e65ccb256059ba504b883017955a9694af8adcbc8ace25cd9c0e13f" gracePeriod=30
Sep 11 17:07:04.627865 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:04.627645    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kube-rbac-proxy" containerID="cri-o://e2e5276dfa11b040185d45f1cdf4fc9b9b118f7a13c1b432f0db22979978d027" gracePeriod=30
Sep 11 17:07:04.660405 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:04.660370    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:04.867067 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:04.867019    2576 generic.go:356] "Generic (PLEG): container finished" podID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerID="18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d" exitCode=0
Sep 11 17:07:04.867259 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:04.867096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" event={"ID":"12700993-9eef-47bb-aa76-fb84a21a3d3f","Type":"ContainerDied","Data":"18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d"}
Sep 11 17:07:04.869435 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:04.869415    2576 generic.go:356] "Generic (PLEG): container finished" podID="e6813771-3140-47aa-b4b4-26b73be882d3" containerID="e2e5276dfa11b040185d45f1cdf4fc9b9b118f7a13c1b432f0db22979978d027" exitCode=2
Sep 11 17:07:04.869510 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:04.869444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" event={"ID":"e6813771-3140-47aa-b4b4-26b73be882d3","Type":"ContainerDied","Data":"e2e5276dfa11b040185d45f1cdf4fc9b9b118f7a13c1b432f0db22979978d027"}
Sep 11 17:07:07.651199 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:07.651109    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Sep 11 17:07:07.656605 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:07.656528    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 11 17:07:07.897407 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:07.897342    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" event={"ID":"e6813771-3140-47aa-b4b4-26b73be882d3","Type":"ContainerDied","Data":"74a8df534e65ccb256059ba504b883017955a9694af8adcbc8ace25cd9c0e13f"}
Sep 11 17:07:07.897600 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:07.897228    2576 generic.go:356] "Generic (PLEG): container finished" podID="e6813771-3140-47aa-b4b4-26b73be882d3" containerID="74a8df534e65ccb256059ba504b883017955a9694af8adcbc8ace25cd9c0e13f" exitCode=0
Sep 11 17:07:09.440140 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.440113    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:07:09.565983 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.565913    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-kube-rbac-proxy-sar-config\") pod \"e6813771-3140-47aa-b4b4-26b73be882d3\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") "
Sep 11 17:07:09.566169 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.566120    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2dzb\" (UniqueName: \"kubernetes.io/projected/e6813771-3140-47aa-b4b4-26b73be882d3-kube-api-access-w2dzb\") pod \"e6813771-3140-47aa-b4b4-26b73be882d3\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") "
Sep 11 17:07:09.566169 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.566163    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6813771-3140-47aa-b4b4-26b73be882d3-kserve-provision-location\") pod \"e6813771-3140-47aa-b4b4-26b73be882d3\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") "
Sep 11 17:07:09.566294 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.566189    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6813771-3140-47aa-b4b4-26b73be882d3-proxy-tls\") pod \"e6813771-3140-47aa-b4b4-26b73be882d3\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") "
Sep 11 17:07:09.566294 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.566210    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-cabundle-cert\") pod \"e6813771-3140-47aa-b4b4-26b73be882d3\" (UID: \"e6813771-3140-47aa-b4b4-26b73be882d3\") "
Sep 11 17:07:09.566395 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.566324    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6813771-3140-47aa-b4b4-26b73be882d3" (UID: "e6813771-3140-47aa-b4b4-26b73be882d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:07:09.566481 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.566463    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:07:09.566723 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.566697    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6813771-3140-47aa-b4b4-26b73be882d3" (UID: "e6813771-3140-47aa-b4b4-26b73be882d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:07:09.568993 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.568958    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6813771-3140-47aa-b4b4-26b73be882d3-kube-api-access-w2dzb" (OuterVolumeSpecName: "kube-api-access-w2dzb") pod "e6813771-3140-47aa-b4b4-26b73be882d3" (UID: "e6813771-3140-47aa-b4b4-26b73be882d3"). InnerVolumeSpecName "kube-api-access-w2dzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:07:09.569375 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.569332    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6813771-3140-47aa-b4b4-26b73be882d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6813771-3140-47aa-b4b4-26b73be882d3" (UID: "e6813771-3140-47aa-b4b4-26b73be882d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:07:09.576464 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.576439    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6813771-3140-47aa-b4b4-26b73be882d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6813771-3140-47aa-b4b4-26b73be882d3" (UID: "e6813771-3140-47aa-b4b4-26b73be882d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:07:09.667263 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.667226    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2dzb\" (UniqueName: \"kubernetes.io/projected/e6813771-3140-47aa-b4b4-26b73be882d3-kube-api-access-w2dzb\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:07:09.667263 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.667253    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6813771-3140-47aa-b4b4-26b73be882d3-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:07:09.667263 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.667263    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6813771-3140-47aa-b4b4-26b73be882d3-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:07:09.667263 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.667272    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6813771-3140-47aa-b4b4-26b73be882d3-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:07:09.800504 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.800465    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:09.839434 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.839404    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"]
Sep 11 17:07:09.839775 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.839746    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container"
Sep 11 17:07:09.839775 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.839763    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container"
Sep 11 17:07:09.840008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.839786    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kube-rbac-proxy"
Sep 11 17:07:09.840008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.839794    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kube-rbac-proxy"
Sep 11 17:07:09.840008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.839804    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="storage-initializer"
Sep 11 17:07:09.840008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.839810    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="storage-initializer"
Sep 11 17:07:09.840008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.839859    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kserve-container"
Sep 11 17:07:09.840008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.839872    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" containerName="kube-rbac-proxy"
Sep 11 17:07:09.844464 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.844445    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:09.847205 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.847178    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 11 17:07:09.847314 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.847231    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 11 17:07:09.854685 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.854664    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"]
Sep 11 17:07:09.909486 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.909437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2" event={"ID":"e6813771-3140-47aa-b4b4-26b73be882d3","Type":"ContainerDied","Data":"e66ad975cb21e860f7e47b93200f436bf0770472493b1c550107e69d5f2cea91"}
Sep 11 17:07:09.909669 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.909523    2576 scope.go:117] "RemoveContainer" containerID="e2e5276dfa11b040185d45f1cdf4fc9b9b118f7a13c1b432f0db22979978d027"
Sep 11 17:07:09.909669 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.909455    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"
Sep 11 17:07:09.934892 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.934864    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"]
Sep 11 17:07:09.939555 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.939527    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2"]
Sep 11 17:07:09.969412 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.969382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:09.969545 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.969415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/927d890a-e23d-408a-b60f-21d08205924c-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:09.969545 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.969443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/927d890a-e23d-408a-b60f-21d08205924c-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:09.969545 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.969500    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcng\" (UniqueName: \"kubernetes.io/projected/927d890a-e23d-408a-b60f-21d08205924c-kube-api-access-rkcng\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:09.969545 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.969538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:09.973146 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:09.973112    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:10.070644 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.070567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.070644 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.070607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/927d890a-e23d-408a-b60f-21d08205924c-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.070644 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.070635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/927d890a-e23d-408a-b60f-21d08205924c-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.070969 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.070745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcng\" (UniqueName: \"kubernetes.io/projected/927d890a-e23d-408a-b60f-21d08205924c-kube-api-access-rkcng\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.070969 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.070782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.071094 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.071051    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/927d890a-e23d-408a-b60f-21d08205924c-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.071392 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.071364    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.071551 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.071527    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.073916 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.073896    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/927d890a-e23d-408a-b60f-21d08205924c-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.080578 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.080554    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcng\" (UniqueName: \"kubernetes.io/projected/927d890a-e23d-408a-b60f-21d08205924c-kube-api-access-rkcng\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.157406 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.157356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:10.604303 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:10.604266    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6813771-3140-47aa-b4b4-26b73be882d3" path="/var/lib/kubelet/pods/e6813771-3140-47aa-b4b4-26b73be882d3/volumes"
Sep 11 17:07:12.027803 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:12.027546    2576 scope.go:117] "RemoveContainer" containerID="74a8df534e65ccb256059ba504b883017955a9694af8adcbc8ace25cd9c0e13f"
Sep 11 17:07:12.125496 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:12.125476    2576 scope.go:117] "RemoveContainer" containerID="eae42b28ebe0cf33de5c5ccf4fcf10d955f65d7b9d19a98c2dee3935302fd00a"
Sep 11 17:07:12.391742 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:12.391716    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"]
Sep 11 17:07:12.395018 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:07:12.394990    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927d890a_e23d_408a_b60f_21d08205924c.slice/crio-583749c37f387e128b122e8bed0bf193eb06f22dedaf9d4ed1e97753ec91975e WatchSource:0}: Error finding container 583749c37f387e128b122e8bed0bf193eb06f22dedaf9d4ed1e97753ec91975e: Status 404 returned error can't find the container with id 583749c37f387e128b122e8bed0bf193eb06f22dedaf9d4ed1e97753ec91975e
Sep 11 17:07:12.922858 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:12.922819    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" event={"ID":"927d890a-e23d-408a-b60f-21d08205924c","Type":"ContainerStarted","Data":"2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c"}
Sep 11 17:07:12.923108 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:12.922887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" event={"ID":"927d890a-e23d-408a-b60f-21d08205924c","Type":"ContainerStarted","Data":"583749c37f387e128b122e8bed0bf193eb06f22dedaf9d4ed1e97753ec91975e"}
Sep 11 17:07:12.925113 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:12.925085    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" event={"ID":"12700993-9eef-47bb-aa76-fb84a21a3d3f","Type":"ContainerStarted","Data":"3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5"}
Sep 11 17:07:12.925250 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:12.925150    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" event={"ID":"12700993-9eef-47bb-aa76-fb84a21a3d3f","Type":"ContainerStarted","Data":"88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb"}
Sep 11 17:07:12.925389 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:12.925370    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:07:12.966440 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:12.966382    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podStartSLOduration=2.73487893 podStartE2EDuration="13.966355891s" totalImagesPullingTime="7.346514965s" totalInitContainerRuntime="3.884961996s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:07:12.96345887 +0000 UTC m=+3026.873260030" watchObservedRunningTime="2026-09-11 17:07:12.966355891 +0000 UTC m=+3026.876157057"
Sep 11 17:07:13.930094 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:13.930062    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:07:13.931586 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:13.931352    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:07:14.934398 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:14.934358    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:07:17.948761 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:17.948724    2576 generic.go:356] "Generic (PLEG): container finished" podID="927d890a-e23d-408a-b60f-21d08205924c" containerID="2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c" exitCode=0
Sep 11 17:07:17.948761 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:17.948763    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" event={"ID":"927d890a-e23d-408a-b60f-21d08205924c","Type":"ContainerDied","Data":"2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c"}
Sep 11 17:07:18.954510 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:18.954469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" event={"ID":"927d890a-e23d-408a-b60f-21d08205924c","Type":"ContainerStarted","Data":"768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614"}
Sep 11 17:07:18.955002 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:18.954538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" event={"ID":"927d890a-e23d-408a-b60f-21d08205924c","Type":"ContainerStarted","Data":"034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1"}
Sep 11 17:07:18.955002 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:18.954784    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:18.979200 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:18.979151    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" podStartSLOduration=5.500458113 podStartE2EDuration="9.979135667s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.478677554s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:07:18.97769339 +0000 UTC m=+3032.887494552" watchObservedRunningTime="2026-09-11 17:07:18.979135667 +0000 UTC m=+3032.888936826"
Sep 11 17:07:19.938827 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:19.938797    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:07:19.939409 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:19.939372    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:07:19.958895 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:19.958867    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:19.960101 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:19.960075    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 11 17:07:20.962452 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:20.962410    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 11 17:07:25.966807 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:25.966777    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:07:25.967471 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:25.967434    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 11 17:07:29.940141 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:29.940103    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:07:31.883566 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:31.883534    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 17:07:31.885801 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:31.885776    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 17:07:31.890735 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:31.890715    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 17:07:31.893011 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:31.892991    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 17:07:35.968043 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:35.968006    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 11 17:07:39.939702 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:39.939614    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:07:45.968061 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:45.968009    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 11 17:07:49.939359 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:49.939321    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:07:55.968181 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:55.968134    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 11 17:07:59.940146 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:07:59.940103    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:08:05.968660 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:05.968633    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:08:09.939762 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:09.939722    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:08:11.246061 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:11.246028    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"]
Sep 11 17:08:11.246453 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:11.246247    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container" containerID="cri-o://034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1" gracePeriod=30
Sep 11 17:08:11.246453 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:11.246304    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kube-rbac-proxy" containerID="cri-o://768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614" gracePeriod=30
Sep 11 17:08:11.254652 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:11.254628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:12.172553 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:12.172520    2576 generic.go:356] "Generic (PLEG): container finished" podID="927d890a-e23d-408a-b60f-21d08205924c" containerID="768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614" exitCode=2
Sep 11 17:08:12.172735 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:12.172588    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" event={"ID":"927d890a-e23d-408a-b60f-21d08205924c","Type":"ContainerDied","Data":"768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614"}
Sep 11 17:08:14.093965 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.093921    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:08:14.177116 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.177024    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-kube-rbac-proxy-sar-config\") pod \"927d890a-e23d-408a-b60f-21d08205924c\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") "
Sep 11 17:08:14.177116 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.177059    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkcng\" (UniqueName: \"kubernetes.io/projected/927d890a-e23d-408a-b60f-21d08205924c-kube-api-access-rkcng\") pod \"927d890a-e23d-408a-b60f-21d08205924c\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") "
Sep 11 17:08:14.177116 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.177113    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/927d890a-e23d-408a-b60f-21d08205924c-kserve-provision-location\") pod \"927d890a-e23d-408a-b60f-21d08205924c\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") "
Sep 11 17:08:14.177361 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.177151    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-cabundle-cert\") pod \"927d890a-e23d-408a-b60f-21d08205924c\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") "
Sep 11 17:08:14.177361 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.177220    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/927d890a-e23d-408a-b60f-21d08205924c-proxy-tls\") pod \"927d890a-e23d-408a-b60f-21d08205924c\" (UID: \"927d890a-e23d-408a-b60f-21d08205924c\") "
Sep 11 17:08:14.177555 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.177526    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "927d890a-e23d-408a-b60f-21d08205924c" (UID: "927d890a-e23d-408a-b60f-21d08205924c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:08:14.177622 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.177572    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "927d890a-e23d-408a-b60f-21d08205924c" (UID: "927d890a-e23d-408a-b60f-21d08205924c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:08:14.179529 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.179501    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/927d890a-e23d-408a-b60f-21d08205924c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "927d890a-e23d-408a-b60f-21d08205924c" (UID: "927d890a-e23d-408a-b60f-21d08205924c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:08:14.179762 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.179734    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927d890a-e23d-408a-b60f-21d08205924c-kube-api-access-rkcng" (OuterVolumeSpecName: "kube-api-access-rkcng") pod "927d890a-e23d-408a-b60f-21d08205924c" (UID: "927d890a-e23d-408a-b60f-21d08205924c"). InnerVolumeSpecName "kube-api-access-rkcng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:08:14.190015 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.189976    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927d890a-e23d-408a-b60f-21d08205924c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "927d890a-e23d-408a-b60f-21d08205924c" (UID: "927d890a-e23d-408a-b60f-21d08205924c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:14.191087 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.191047    2576 generic.go:356] "Generic (PLEG): container finished" podID="927d890a-e23d-408a-b60f-21d08205924c" containerID="034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1" exitCode=0
Sep 11 17:08:14.191226 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.191137    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"
Sep 11 17:08:14.191226 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.191131    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" event={"ID":"927d890a-e23d-408a-b60f-21d08205924c","Type":"ContainerDied","Data":"034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1"}
Sep 11 17:08:14.191361 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.191315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9" event={"ID":"927d890a-e23d-408a-b60f-21d08205924c","Type":"ContainerDied","Data":"583749c37f387e128b122e8bed0bf193eb06f22dedaf9d4ed1e97753ec91975e"}
Sep 11 17:08:14.191361 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.191357    2576 scope.go:117] "RemoveContainer" containerID="768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614"
Sep 11 17:08:14.201606 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.201589    2576 scope.go:117] "RemoveContainer" containerID="034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1"
Sep 11 17:08:14.210295 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.210269    2576 scope.go:117] "RemoveContainer" containerID="2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c"
Sep 11 17:08:14.217405 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.217379    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"]
Sep 11 17:08:14.218360 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.218339    2576 scope.go:117] "RemoveContainer" containerID="768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614"
Sep 11 17:08:14.218646 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:08:14.218620    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614\": container with ID starting with 768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614 not found: ID does not exist" containerID="768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614"
Sep 11 17:08:14.218699 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.218657    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614"} err="failed to get container status \"768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614\": rpc error: code = NotFound desc = could not find container \"768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614\": container with ID starting with 768998e695dcc84f85e1bdcec200781990131c6f6812137a6abf04cc0428a614 not found: ID does not exist"
Sep 11 17:08:14.218699 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.218676    2576 scope.go:117] "RemoveContainer" containerID="034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1"
Sep 11 17:08:14.218902 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:08:14.218885    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1\": container with ID starting with 034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1 not found: ID does not exist" containerID="034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1"
Sep 11 17:08:14.218965 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.218907    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1"} err="failed to get container status \"034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1\": rpc error: code = NotFound desc = could not find container \"034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1\": container with ID starting with 034957f4670ad4cecb037ff46af19d4aa93451141e51ee6b46e67080af6055d1 not found: ID does not exist"
Sep 11 17:08:14.218965 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.218923    2576 scope.go:117] "RemoveContainer" containerID="2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c"
Sep 11 17:08:14.219389 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:08:14.219371    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c\": container with ID starting with 2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c not found: ID does not exist" containerID="2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c"
Sep 11 17:08:14.219451 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.219390    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c"} err="failed to get container status \"2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c\": rpc error: code = NotFound desc = could not find container \"2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c\": container with ID starting with 2bf292efc650f8155576397a955d90cf8bcbf8ba95b2c5477d3ffb1800e3e24c not found: ID does not exist"
Sep 11 17:08:14.222470 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.222451    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9"]
Sep 11 17:08:14.278241 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.278205    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:14.278241 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.278234    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/927d890a-e23d-408a-b60f-21d08205924c-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:14.278241 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.278243    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/927d890a-e23d-408a-b60f-21d08205924c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:14.278469 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.278254    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkcng\" (UniqueName: \"kubernetes.io/projected/927d890a-e23d-408a-b60f-21d08205924c-kube-api-access-rkcng\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:14.278469 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.278264    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/927d890a-e23d-408a-b60f-21d08205924c-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:14.603646 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:14.603613    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="927d890a-e23d-408a-b60f-21d08205924c" path="/var/lib/kubelet/pods/927d890a-e23d-408a-b60f-21d08205924c/volumes"
Sep 11 17:08:16.379831 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:16.379799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:16.694620 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:16.694591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:19.939288 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:19.939250    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:08:29.940223 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:29.940183    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 11 17:08:39.940051 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:39.940022    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:08:50.418591 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:50.418550    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"]
Sep 11 17:08:50.419147 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:50.418849    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container" containerID="cri-o://88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb" gracePeriod=30
Sep 11 17:08:50.419147 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:50.418955    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kube-rbac-proxy" containerID="cri-o://3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5" gracePeriod=30
Sep 11 17:08:50.445375 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:50.445346    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:51.331108 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:51.331077    2576 generic.go:356] "Generic (PLEG): container finished" podID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerID="3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5" exitCode=2
Sep 11 17:08:51.331283 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:51.331151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" event={"ID":"12700993-9eef-47bb-aa76-fb84a21a3d3f","Type":"ContainerDied","Data":"3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5"}
Sep 11 17:08:54.055118 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.055090    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:08:54.086313 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.086288    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12700993-9eef-47bb-aa76-fb84a21a3d3f-kserve-provision-location\") pod \"12700993-9eef-47bb-aa76-fb84a21a3d3f\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") "
Sep 11 17:08:54.086469 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.086322    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-rbac-proxy-sar-config\") pod \"12700993-9eef-47bb-aa76-fb84a21a3d3f\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") "
Sep 11 17:08:54.086469 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.086338    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-cabundle-cert\") pod \"12700993-9eef-47bb-aa76-fb84a21a3d3f\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") "
Sep 11 17:08:54.086469 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.086374    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29p87\" (UniqueName: \"kubernetes.io/projected/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-api-access-29p87\") pod \"12700993-9eef-47bb-aa76-fb84a21a3d3f\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") "
Sep 11 17:08:54.086640 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.086479    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12700993-9eef-47bb-aa76-fb84a21a3d3f-proxy-tls\") pod \"12700993-9eef-47bb-aa76-fb84a21a3d3f\" (UID: \"12700993-9eef-47bb-aa76-fb84a21a3d3f\") "
Sep 11 17:08:54.086704 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.086652    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12700993-9eef-47bb-aa76-fb84a21a3d3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12700993-9eef-47bb-aa76-fb84a21a3d3f" (UID: "12700993-9eef-47bb-aa76-fb84a21a3d3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:54.086704 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.086686    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12700993-9eef-47bb-aa76-fb84a21a3d3f" (UID: "12700993-9eef-47bb-aa76-fb84a21a3d3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:08:54.086810 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.086714    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12700993-9eef-47bb-aa76-fb84a21a3d3f" (UID: "12700993-9eef-47bb-aa76-fb84a21a3d3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:08:54.086810 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.086725    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12700993-9eef-47bb-aa76-fb84a21a3d3f-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:54.088799 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.088771    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12700993-9eef-47bb-aa76-fb84a21a3d3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12700993-9eef-47bb-aa76-fb84a21a3d3f" (UID: "12700993-9eef-47bb-aa76-fb84a21a3d3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:08:54.089058 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.089037    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-api-access-29p87" (OuterVolumeSpecName: "kube-api-access-29p87") pod "12700993-9eef-47bb-aa76-fb84a21a3d3f" (UID: "12700993-9eef-47bb-aa76-fb84a21a3d3f"). InnerVolumeSpecName "kube-api-access-29p87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:08:54.187192 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.187111    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:54.187192 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.187137    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12700993-9eef-47bb-aa76-fb84a21a3d3f-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:54.187192 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.187147    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29p87\" (UniqueName: \"kubernetes.io/projected/12700993-9eef-47bb-aa76-fb84a21a3d3f-kube-api-access-29p87\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:54.187192 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.187159    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12700993-9eef-47bb-aa76-fb84a21a3d3f-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:08:54.343687 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.343654    2576 generic.go:356] "Generic (PLEG): container finished" podID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerID="88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb" exitCode=0
Sep 11 17:08:54.343867 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.343691    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" event={"ID":"12700993-9eef-47bb-aa76-fb84a21a3d3f","Type":"ContainerDied","Data":"88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb"}
Sep 11 17:08:54.343867 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.343717    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"
Sep 11 17:08:54.343867 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.343752    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb" event={"ID":"12700993-9eef-47bb-aa76-fb84a21a3d3f","Type":"ContainerDied","Data":"5418d8d415c60916da5bfc3926c8e1d2af43c6cc915371496685929df586652e"}
Sep 11 17:08:54.343867 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.343781    2576 scope.go:117] "RemoveContainer" containerID="3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5"
Sep 11 17:08:54.354457 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.354401    2576 scope.go:117] "RemoveContainer" containerID="88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb"
Sep 11 17:08:54.362154 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.362139    2576 scope.go:117] "RemoveContainer" containerID="18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d"
Sep 11 17:08:54.369411 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.369394    2576 scope.go:117] "RemoveContainer" containerID="3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5"
Sep 11 17:08:54.369695 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:08:54.369671    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5\": container with ID starting with 3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5 not found: ID does not exist" containerID="3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5"
Sep 11 17:08:54.369769 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.369704    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5"} err="failed to get container status \"3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5\": rpc error: code = NotFound desc = could not find container \"3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5\": container with ID starting with 3d4b589ea9162bb457f2a575062c94b6c761d5d72b8bffc909caa02097bf04b5 not found: ID does not exist"
Sep 11 17:08:54.369769 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.369722    2576 scope.go:117] "RemoveContainer" containerID="88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb"
Sep 11 17:08:54.369995 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:08:54.369970    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb\": container with ID starting with 88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb not found: ID does not exist" containerID="88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb"
Sep 11 17:08:54.370065 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.370002    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb"} err="failed to get container status \"88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb\": rpc error: code = NotFound desc = could not find container \"88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb\": container with ID starting with 88f921ee1347db5164cdc4e20d7d2d23a793eadfa3df0e6464cae483e05476bb not found: ID does not exist"
Sep 11 17:08:54.370065 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.370023    2576 scope.go:117] "RemoveContainer" containerID="18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d"
Sep 11 17:08:54.370307 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:08:54.370291    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d\": container with ID starting with 18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d not found: ID does not exist" containerID="18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d"
Sep 11 17:08:54.370372 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.370310    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d"} err="failed to get container status \"18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d\": rpc error: code = NotFound desc = could not find container \"18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d\": container with ID starting with 18553df5490c1cd25270aeaf5e3bc918122a8b3cddefa28b58786b1269a8463d not found: ID does not exist"
Sep 11 17:08:54.370630 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.370610    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"]
Sep 11 17:08:54.378006 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.377986    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb"]
Sep 11 17:08:54.602324 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:54.602290    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" path="/var/lib/kubelet/pods/12700993-9eef-47bb-aa76-fb84a21a3d3f/volumes"
Sep 11 17:08:55.570092 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.570044    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:55.612466 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612432    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"]
Sep 11 17:08:55.612703 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612692    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kube-rbac-proxy"
Sep 11 17:08:55.612756 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612705    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kube-rbac-proxy"
Sep 11 17:08:55.612756 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612714    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container"
Sep 11 17:08:55.612756 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612719    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container"
Sep 11 17:08:55.612756 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612727    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kube-rbac-proxy"
Sep 11 17:08:55.612756 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612733    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kube-rbac-proxy"
Sep 11 17:08:55.612756 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612747    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="storage-initializer"
Sep 11 17:08:55.612756 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612754    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="storage-initializer"
Sep 11 17:08:55.613012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612762    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="storage-initializer"
Sep 11 17:08:55.613012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612767    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="storage-initializer"
Sep 11 17:08:55.613012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612776    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container"
Sep 11 17:08:55.613012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612781    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container"
Sep 11 17:08:55.613012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612835    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kube-rbac-proxy"
Sep 11 17:08:55.613012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612843    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="12700993-9eef-47bb-aa76-fb84a21a3d3f" containerName="kserve-container"
Sep 11 17:08:55.613012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612849    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kserve-container"
Sep 11 17:08:55.613012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.612856    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="927d890a-e23d-408a-b60f-21d08205924c" containerName="kube-rbac-proxy"
Sep 11 17:08:55.617167 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.617144    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.619796 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.619770    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 17:08:55.619796 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.619788    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 17:08:55.620914 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.620893    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:08:55.621095 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.620894    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 17:08:55.621095 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.621029    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 17:08:55.621383 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.620979    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 11 17:08:55.621383 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.620976    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 11 17:08:55.631547 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.631526    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"]
Sep 11 17:08:55.697530 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.697495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.697530 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.697529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.697749 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.697547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjlz9\" (UniqueName: \"kubernetes.io/projected/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-api-access-mjlz9\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.697749 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.697707    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.697839 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.697756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.721954 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.721906    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:55.798155 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.798114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.798155 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.798158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjlz9\" (UniqueName: \"kubernetes.io/projected/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-api-access-mjlz9\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.798421 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.798202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.798421 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.798234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.798421 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.798280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.798658 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.798634    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.798889 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.798865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.798977 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.798896    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.800766 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.800747    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.809403 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.809378    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjlz9\" (UniqueName: \"kubernetes.io/projected/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-api-access-mjlz9\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-286wj\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:55.928339 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:55.928251    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:08:56.200634 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:56.200608    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"]
Sep 11 17:08:56.203131 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:08:56.203101    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d103f31_5c4d_416c_b2c7_7bbef6ae3353.slice/crio-e28a78a61c6ad2f0ec79a18421c67933c0a7d2025f41e32c101661b1fc8a5bc4 WatchSource:0}: Error finding container e28a78a61c6ad2f0ec79a18421c67933c0a7d2025f41e32c101661b1fc8a5bc4: Status 404 returned error can't find the container with id e28a78a61c6ad2f0ec79a18421c67933c0a7d2025f41e32c101661b1fc8a5bc4
Sep 11 17:08:56.355353 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:56.355317    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" event={"ID":"6d103f31-5c4d-416c-b2c7-7bbef6ae3353","Type":"ContainerStarted","Data":"ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140"}
Sep 11 17:08:56.358515 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:08:56.355757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" event={"ID":"6d103f31-5c4d-416c-b2c7-7bbef6ae3353","Type":"ContainerStarted","Data":"e28a78a61c6ad2f0ec79a18421c67933c0a7d2025f41e32c101661b1fc8a5bc4"}
Sep 11 17:09:00.371567 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:00.371533    2576 generic.go:356] "Generic (PLEG): container finished" podID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerID="ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140" exitCode=0
Sep 11 17:09:00.371979 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:00.371605    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" event={"ID":"6d103f31-5c4d-416c-b2c7-7bbef6ae3353","Type":"ContainerDied","Data":"ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140"}
Sep 11 17:09:01.378552 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:01.378516    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" event={"ID":"6d103f31-5c4d-416c-b2c7-7bbef6ae3353","Type":"ContainerStarted","Data":"11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453"}
Sep 11 17:09:01.379003 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:01.378575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" event={"ID":"6d103f31-5c4d-416c-b2c7-7bbef6ae3353","Type":"ContainerStarted","Data":"a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f"}
Sep 11 17:09:01.379003 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:01.378830    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:09:01.379003 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:01.378972    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:09:01.380293 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:01.380265    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:09:01.401644 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:01.401604    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podStartSLOduration=2.459568687 podStartE2EDuration="6.401589153s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.942020466s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:09:01.400052584 +0000 UTC m=+3135.309853750" watchObservedRunningTime="2026-09-11 17:09:01.401589153 +0000 UTC m=+3135.311390315"
Sep 11 17:09:02.382468 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:02.382422    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:09:07.387573 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:07.387544    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:09:07.388076 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:07.388050    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:09:17.388735 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:17.388696    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:09:27.388608 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:27.388561    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:09:28.703144 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:28.703106    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:37.388857 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:37.388820    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:09:47.388483 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:47.388443    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:09:57.388603 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:57.388557    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:09:57.517431 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:09:57.517398    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:07.389020 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:07.388980    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:10:17.388510 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:17.388465    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 11 17:10:26.604005 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:26.603974    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:10:36.959442 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:36.959407    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"]
Sep 11 17:10:36.960040 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:36.959715    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container" containerID="cri-o://a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f" gracePeriod=30
Sep 11 17:10:36.960040 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:36.959895    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kube-rbac-proxy" containerID="cri-o://11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453" gracePeriod=30
Sep 11 17:10:36.974924 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:36.974903    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:37.383747 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:37.383657    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Sep 11 17:10:37.743377 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:37.743343    2576 generic.go:356] "Generic (PLEG): container finished" podID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerID="11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453" exitCode=2
Sep 11 17:10:37.743540 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:37.743384    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" event={"ID":"6d103f31-5c4d-416c-b2c7-7bbef6ae3353","Type":"ContainerDied","Data":"11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453"}
Sep 11 17:10:40.504366 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.504341    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:10:40.597049 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.596966    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kserve-provision-location\") pod \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") "
Sep 11 17:10:40.597049 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.597005    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-rbac-proxy-sar-config\") pod \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") "
Sep 11 17:10:40.597049 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.597033    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjlz9\" (UniqueName: \"kubernetes.io/projected/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-api-access-mjlz9\") pod \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") "
Sep 11 17:10:40.597343 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.597060    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-cabundle-cert\") pod \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") "
Sep 11 17:10:40.597343 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.597222    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-proxy-tls\") pod \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\" (UID: \"6d103f31-5c4d-416c-b2c7-7bbef6ae3353\") "
Sep 11 17:10:40.597458 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.597359    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d103f31-5c4d-416c-b2c7-7bbef6ae3353" (UID: "6d103f31-5c4d-416c-b2c7-7bbef6ae3353"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:10:40.597458 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.597411    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d103f31-5c4d-416c-b2c7-7bbef6ae3353" (UID: "6d103f31-5c4d-416c-b2c7-7bbef6ae3353"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:10:40.597458 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.597419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d103f31-5c4d-416c-b2c7-7bbef6ae3353" (UID: "6d103f31-5c4d-416c-b2c7-7bbef6ae3353"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:10:40.597595 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.597461    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:10:40.599427 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.599395    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d103f31-5c4d-416c-b2c7-7bbef6ae3353" (UID: "6d103f31-5c4d-416c-b2c7-7bbef6ae3353"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:10:40.599542 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.599463    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-api-access-mjlz9" (OuterVolumeSpecName: "kube-api-access-mjlz9") pod "6d103f31-5c4d-416c-b2c7-7bbef6ae3353" (UID: "6d103f31-5c4d-416c-b2c7-7bbef6ae3353"). InnerVolumeSpecName "kube-api-access-mjlz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:10:40.698088 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.698057    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:10:40.698088 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.698083    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjlz9\" (UniqueName: \"kubernetes.io/projected/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-kube-api-access-mjlz9\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:10:40.698285 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.698098    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:10:40.698285 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.698111    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d103f31-5c4d-416c-b2c7-7bbef6ae3353-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:10:40.755445 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.755414    2576 generic.go:356] "Generic (PLEG): container finished" podID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerID="a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f" exitCode=0
Sep 11 17:10:40.755563 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.755465    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"
Sep 11 17:10:40.755563 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.755473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" event={"ID":"6d103f31-5c4d-416c-b2c7-7bbef6ae3353","Type":"ContainerDied","Data":"a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f"}
Sep 11 17:10:40.755563 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.755530    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj" event={"ID":"6d103f31-5c4d-416c-b2c7-7bbef6ae3353","Type":"ContainerDied","Data":"e28a78a61c6ad2f0ec79a18421c67933c0a7d2025f41e32c101661b1fc8a5bc4"}
Sep 11 17:10:40.755563 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.755560    2576 scope.go:117] "RemoveContainer" containerID="11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453"
Sep 11 17:10:40.763426 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.763384    2576 scope.go:117] "RemoveContainer" containerID="a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f"
Sep 11 17:10:40.770625 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.770608    2576 scope.go:117] "RemoveContainer" containerID="ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140"
Sep 11 17:10:40.774155 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.774135    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"]
Sep 11 17:10:40.778536 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.778520    2576 scope.go:117] "RemoveContainer" containerID="11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453"
Sep 11 17:10:40.778602 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.778520    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj"]
Sep 11 17:10:40.778791 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:10:40.778775    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453\": container with ID starting with 11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453 not found: ID does not exist" containerID="11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453"
Sep 11 17:10:40.778850 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.778798    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453"} err="failed to get container status \"11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453\": rpc error: code = NotFound desc = could not find container \"11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453\": container with ID starting with 11c18311392de3bc4e30a0a1c33b1eea9ef95d8f5b3d26825bc7ba5c9dc6c453 not found: ID does not exist"
Sep 11 17:10:40.778850 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.778813    2576 scope.go:117] "RemoveContainer" containerID="a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f"
Sep 11 17:10:40.779104 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:10:40.779086    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f\": container with ID starting with a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f not found: ID does not exist" containerID="a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f"
Sep 11 17:10:40.779174 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.779111    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f"} err="failed to get container status \"a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f\": rpc error: code = NotFound desc = could not find container \"a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f\": container with ID starting with a975e925b369c4e4e634103570795c2f9e4c5c8e2a3c112e08e526c2268dc57f not found: ID does not exist"
Sep 11 17:10:40.779174 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.779128    2576 scope.go:117] "RemoveContainer" containerID="ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140"
Sep 11 17:10:40.779397 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:10:40.779381    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140\": container with ID starting with ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140 not found: ID does not exist" containerID="ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140"
Sep 11 17:10:40.779447 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:40.779404    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140"} err="failed to get container status \"ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140\": rpc error: code = NotFound desc = could not find container \"ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140\": container with ID starting with ca8e80a77920d64ef127212752d2b3a785c99a87ffca10f9c05a1022ff50e140 not found: ID does not exist"
Sep 11 17:10:42.085664 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.085634    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:42.152985 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.152954    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"]
Sep 11 17:10:42.153246 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.153234    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container"
Sep 11 17:10:42.153289 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.153248    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container"
Sep 11 17:10:42.153289 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.153257    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kube-rbac-proxy"
Sep 11 17:10:42.153289 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.153263    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kube-rbac-proxy"
Sep 11 17:10:42.153289 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.153270    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="storage-initializer"
Sep 11 17:10:42.153289 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.153275    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="storage-initializer"
Sep 11 17:10:42.153442 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.153324    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kube-rbac-proxy"
Sep 11 17:10:42.153442 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.153339    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" containerName="kserve-container"
Sep 11 17:10:42.157734 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.157715    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.160482 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.160462    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-932c75-kube-rbac-proxy-sar-config\""
Sep 11 17:10:42.160571 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.160522    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 17:10:42.160841 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.160824    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 17:10:42.161684 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.161661    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:10:42.161684 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.161678    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 17:10:42.161907 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.161702    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 17:10:42.161907 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.161670    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-932c75-predictor-serving-cert\""
Sep 11 17:10:42.167067 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.167037    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"]
Sep 11 17:10:42.208021 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.207989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-cabundle-cert\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.208021 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.208027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gcw\" (UniqueName: \"kubernetes.io/projected/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-api-access-q8gcw\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.208244 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.208046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b8a661-54db-4aa6-a232-315cdddeffa9-proxy-tls\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.208244 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.208131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.208244 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.208199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b8a661-54db-4aa6-a232-315cdddeffa9-kserve-provision-location\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.309319 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.309279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-cabundle-cert\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.309533 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.309330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gcw\" (UniqueName: \"kubernetes.io/projected/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-api-access-q8gcw\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.309533 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.309358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b8a661-54db-4aa6-a232-315cdddeffa9-proxy-tls\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.309533 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.309389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.309533 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.309425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b8a661-54db-4aa6-a232-315cdddeffa9-kserve-provision-location\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.309533 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:10:42.309518    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-932c75-predictor-serving-cert: secret "isvc-primary-932c75-predictor-serving-cert" not found
Sep 11 17:10:42.309807 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:10:42.309605    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b8a661-54db-4aa6-a232-315cdddeffa9-proxy-tls podName:01b8a661-54db-4aa6-a232-315cdddeffa9 nodeName:}" failed. No retries permitted until 2026-09-11 17:10:42.80958236 +0000 UTC m=+3236.719383500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/01b8a661-54db-4aa6-a232-315cdddeffa9-proxy-tls") pod "isvc-primary-932c75-predictor-bd6c947bf-bhkvw" (UID: "01b8a661-54db-4aa6-a232-315cdddeffa9") : secret "isvc-primary-932c75-predictor-serving-cert" not found
Sep 11 17:10:42.309868 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.309820    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b8a661-54db-4aa6-a232-315cdddeffa9-kserve-provision-location\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.310047 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.310028    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-cabundle-cert\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.310094 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.310061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.320288 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.320268    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gcw\" (UniqueName: \"kubernetes.io/projected/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-api-access-q8gcw\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.461552 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.461502    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:42.602793 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.602765    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d103f31-5c4d-416c-b2c7-7bbef6ae3353" path="/var/lib/kubelet/pods/6d103f31-5c4d-416c-b2c7-7bbef6ae3353/volumes"
Sep 11 17:10:42.813972 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.813877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b8a661-54db-4aa6-a232-315cdddeffa9-proxy-tls\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:42.816180 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:42.816162    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b8a661-54db-4aa6-a232-315cdddeffa9-proxy-tls\") pod \"isvc-primary-932c75-predictor-bd6c947bf-bhkvw\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") " pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:43.069987 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:43.069883    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:43.342743 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:43.342711    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"]
Sep 11 17:10:43.345363 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:10:43.345338    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b8a661_54db_4aa6_a232_315cdddeffa9.slice/crio-eb7b4bb65b2f3b0dce9faeb842f793c5b870319123db6cb62ad49dd9bab1c7c3 WatchSource:0}: Error finding container eb7b4bb65b2f3b0dce9faeb842f793c5b870319123db6cb62ad49dd9bab1c7c3: Status 404 returned error can't find the container with id eb7b4bb65b2f3b0dce9faeb842f793c5b870319123db6cb62ad49dd9bab1c7c3
Sep 11 17:10:43.770501 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:43.770462    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" event={"ID":"01b8a661-54db-4aa6-a232-315cdddeffa9","Type":"ContainerStarted","Data":"8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218"}
Sep 11 17:10:43.770674 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:43.770533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" event={"ID":"01b8a661-54db-4aa6-a232-315cdddeffa9","Type":"ContainerStarted","Data":"eb7b4bb65b2f3b0dce9faeb842f793c5b870319123db6cb62ad49dd9bab1c7c3"}
Sep 11 17:10:47.786894 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:47.786860    2576 generic.go:356] "Generic (PLEG): container finished" podID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerID="8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218" exitCode=0
Sep 11 17:10:47.787281 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:47.786955    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" event={"ID":"01b8a661-54db-4aa6-a232-315cdddeffa9","Type":"ContainerDied","Data":"8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218"}
Sep 11 17:10:48.793281 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:48.793248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" event={"ID":"01b8a661-54db-4aa6-a232-315cdddeffa9","Type":"ContainerStarted","Data":"1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd"}
Sep 11 17:10:48.793705 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:48.793306    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" event={"ID":"01b8a661-54db-4aa6-a232-315cdddeffa9","Type":"ContainerStarted","Data":"3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947"}
Sep 11 17:10:48.793705 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:48.793497    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:48.817542 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:48.817494    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podStartSLOduration=2.939954821 podStartE2EDuration="6.817478609s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877523788s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:10:48.81604328 +0000 UTC m=+3242.725844441" watchObservedRunningTime="2026-09-11 17:10:48.817478609 +0000 UTC m=+3242.727279771"
Sep 11 17:10:49.797106 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:49.797062    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:49.798399 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:49.798362    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:10:50.801049 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:50.801009    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:10:55.806080 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:55.806050    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:10:55.806588 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:55.806559    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:10:58.182889 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:10:58.182854    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:11:05.807498 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:11:05.807405    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:11:15.807420 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:11:15.807379    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:11:25.807180 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:11:25.807138    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:11:35.806825 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:11:35.806779    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:11:45.806857 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:11:45.806813    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:11:55.807159 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:11:55.807117    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:12:05.807841 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:05.807806    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:12:27.478253 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.478210    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"]
Sep 11 17:12:27.481585 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.481568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.484152 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.484128    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-932c75-predictor-serving-cert\""
Sep 11 17:12:27.484288 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.484169    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-932c75-dockercfg-z5d9x\""
Sep 11 17:12:27.484441 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.484428    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-932c75-kube-rbac-proxy-sar-config\""
Sep 11 17:12:27.484493 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.484468    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-932c75\""
Sep 11 17:12:27.497785 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.497759    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"]
Sep 11 17:12:27.627772 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.627739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.627772 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.627777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-cabundle-cert\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.628012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.627799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9999eec3-90ad-49a1-85fa-c21c230c2653-kserve-provision-location\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.628012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.627871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9999eec3-90ad-49a1-85fa-c21c230c2653-proxy-tls\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.628012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.627914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkz2\" (UniqueName: \"kubernetes.io/projected/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-api-access-zdkz2\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.728631 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.728545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkz2\" (UniqueName: \"kubernetes.io/projected/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-api-access-zdkz2\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.728631 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.728595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.728631 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.728631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-cabundle-cert\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.728877 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.728660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9999eec3-90ad-49a1-85fa-c21c230c2653-kserve-provision-location\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.728877 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.728791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9999eec3-90ad-49a1-85fa-c21c230c2653-proxy-tls\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.729015 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:12:27.728994    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-932c75-predictor-serving-cert: secret "isvc-secondary-932c75-predictor-serving-cert" not found
Sep 11 17:12:27.729083 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:12:27.729071    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9999eec3-90ad-49a1-85fa-c21c230c2653-proxy-tls podName:9999eec3-90ad-49a1-85fa-c21c230c2653 nodeName:}" failed. No retries permitted until 2026-09-11 17:12:28.229051379 +0000 UTC m=+3342.138852519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9999eec3-90ad-49a1-85fa-c21c230c2653-proxy-tls") pod "isvc-secondary-932c75-predictor-76c5bfcd-xvcc9" (UID: "9999eec3-90ad-49a1-85fa-c21c230c2653") : secret "isvc-secondary-932c75-predictor-serving-cert" not found
Sep 11 17:12:27.729138 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.729091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9999eec3-90ad-49a1-85fa-c21c230c2653-kserve-provision-location\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.729300 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.729283    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-cabundle-cert\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.729438 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.729420    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:27.740859 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:27.740831    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkz2\" (UniqueName: \"kubernetes.io/projected/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-api-access-zdkz2\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:28.234301 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:28.234258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9999eec3-90ad-49a1-85fa-c21c230c2653-proxy-tls\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:28.236791 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:28.236760    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9999eec3-90ad-49a1-85fa-c21c230c2653-proxy-tls\") pod \"isvc-secondary-932c75-predictor-76c5bfcd-xvcc9\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") " pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:28.286263 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:28.286234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:28.391617 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:28.391587    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:28.669167 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:28.669118    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"]
Sep 11 17:12:28.671376 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:12:28.671348    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9999eec3_90ad_49a1_85fa_c21c230c2653.slice/crio-d685aeb0c362be173653677a335c484acdf622cdbc970daa33f0ec949c5360e2 WatchSource:0}: Error finding container d685aeb0c362be173653677a335c484acdf622cdbc970daa33f0ec949c5360e2: Status 404 returned error can't find the container with id d685aeb0c362be173653677a335c484acdf622cdbc970daa33f0ec949c5360e2
Sep 11 17:12:29.179868 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:29.179828    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9" event={"ID":"9999eec3-90ad-49a1-85fa-c21c230c2653","Type":"ContainerStarted","Data":"d8fa1fac558ee8a1ff63e90e4ab914bc0a4eecfb959f83bb6f3d756223a59e7b"}
Sep 11 17:12:29.180062 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:29.179899    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9" event={"ID":"9999eec3-90ad-49a1-85fa-c21c230c2653","Type":"ContainerStarted","Data":"d685aeb0c362be173653677a335c484acdf622cdbc970daa33f0ec949c5360e2"}
Sep 11 17:12:29.255344 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:29.255312    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:31.914613 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:31.914583    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 17:12:31.917561 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:31.917539    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 17:12:31.921466 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:31.921436    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 17:12:31.923990 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:31.923975    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 17:12:33.195744 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:33.195722    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_9999eec3-90ad-49a1-85fa-c21c230c2653/storage-initializer/0.log"
Sep 11 17:12:33.196137 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:33.195759    2576 generic.go:356] "Generic (PLEG): container finished" podID="9999eec3-90ad-49a1-85fa-c21c230c2653" containerID="d8fa1fac558ee8a1ff63e90e4ab914bc0a4eecfb959f83bb6f3d756223a59e7b" exitCode=1
Sep 11 17:12:33.196137 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:33.195826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9" event={"ID":"9999eec3-90ad-49a1-85fa-c21c230c2653","Type":"ContainerDied","Data":"d8fa1fac558ee8a1ff63e90e4ab914bc0a4eecfb959f83bb6f3d756223a59e7b"}
Sep 11 17:12:34.201450 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:34.201415    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_9999eec3-90ad-49a1-85fa-c21c230c2653/storage-initializer/0.log"
Sep 11 17:12:34.201952 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:34.201479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9" event={"ID":"9999eec3-90ad-49a1-85fa-c21c230c2653","Type":"ContainerStarted","Data":"6bd11d94005271442346296f4cfb70fc3c7b6c39b3844cfab66d77b890253a89"}
Sep 11 17:12:39.222867 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:39.222793    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_9999eec3-90ad-49a1-85fa-c21c230c2653/storage-initializer/1.log"
Sep 11 17:12:39.223310 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:39.223245    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_9999eec3-90ad-49a1-85fa-c21c230c2653/storage-initializer/0.log"
Sep 11 17:12:39.223310 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:39.223277    2576 generic.go:356] "Generic (PLEG): container finished" podID="9999eec3-90ad-49a1-85fa-c21c230c2653" containerID="6bd11d94005271442346296f4cfb70fc3c7b6c39b3844cfab66d77b890253a89" exitCode=1
Sep 11 17:12:39.223389 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:39.223319    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9" event={"ID":"9999eec3-90ad-49a1-85fa-c21c230c2653","Type":"ContainerDied","Data":"6bd11d94005271442346296f4cfb70fc3c7b6c39b3844cfab66d77b890253a89"}
Sep 11 17:12:39.223389 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:39.223376    2576 scope.go:117] "RemoveContainer" containerID="d8fa1fac558ee8a1ff63e90e4ab914bc0a4eecfb959f83bb6f3d756223a59e7b"
Sep 11 17:12:39.223822 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:39.223799    2576 scope.go:117] "RemoveContainer" containerID="d8fa1fac558ee8a1ff63e90e4ab914bc0a4eecfb959f83bb6f3d756223a59e7b"
Sep 11 17:12:39.233905 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:12:39.233874    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_kserve-ci-e2e-test(9999eec3-90ad-49a1-85fa-c21c230c2653)\"" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9" podUID="9999eec3-90ad-49a1-85fa-c21c230c2653"
Sep 11 17:12:39.234187 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:12:39.234161    2576 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_kserve-ci-e2e-test_9999eec3-90ad-49a1-85fa-c21c230c2653_0 in pod sandbox d685aeb0c362be173653677a335c484acdf622cdbc970daa33f0ec949c5360e2 from index: no such id: 'd8fa1fac558ee8a1ff63e90e4ab914bc0a4eecfb959f83bb6f3d756223a59e7b'" containerID="d8fa1fac558ee8a1ff63e90e4ab914bc0a4eecfb959f83bb6f3d756223a59e7b"
Sep 11 17:12:39.234247 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:39.234196    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fa1fac558ee8a1ff63e90e4ab914bc0a4eecfb959f83bb6f3d756223a59e7b"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_kserve-ci-e2e-test_9999eec3-90ad-49a1-85fa-c21c230c2653_0 in pod sandbox d685aeb0c362be173653677a335c484acdf622cdbc970daa33f0ec949c5360e2 from index: no such id: 'd8fa1fac558ee8a1ff63e90e4ab914bc0a4eecfb959f83bb6f3d756223a59e7b'"
Sep 11 17:12:40.228581 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:40.228551    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_9999eec3-90ad-49a1-85fa-c21c230c2653/storage-initializer/1.log"
Sep 11 17:12:45.462588 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.462555    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"]
Sep 11 17:12:45.488097 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.488056    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:45.507076 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.507036    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"]
Sep 11 17:12:45.507314 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.507285    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" containerID="cri-o://3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947" gracePeriod=30
Sep 11 17:12:45.507427 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.507318    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kube-rbac-proxy" containerID="cri-o://1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd" gracePeriod=30
Sep 11 17:12:45.584212 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.584178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:45.626062 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.626040    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_9999eec3-90ad-49a1-85fa-c21c230c2653/storage-initializer/1.log"
Sep 11 17:12:45.628127 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.628110    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:45.765890 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.765805    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-rbac-proxy-sar-config\") pod \"9999eec3-90ad-49a1-85fa-c21c230c2653\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") "
Sep 11 17:12:45.765890 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.765840    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9999eec3-90ad-49a1-85fa-c21c230c2653-proxy-tls\") pod \"9999eec3-90ad-49a1-85fa-c21c230c2653\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") "
Sep 11 17:12:45.765890 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.765871    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-cabundle-cert\") pod \"9999eec3-90ad-49a1-85fa-c21c230c2653\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") "
Sep 11 17:12:45.766217 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.765908    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9999eec3-90ad-49a1-85fa-c21c230c2653-kserve-provision-location\") pod \"9999eec3-90ad-49a1-85fa-c21c230c2653\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") "
Sep 11 17:12:45.766217 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.765957    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdkz2\" (UniqueName: \"kubernetes.io/projected/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-api-access-zdkz2\") pod \"9999eec3-90ad-49a1-85fa-c21c230c2653\" (UID: \"9999eec3-90ad-49a1-85fa-c21c230c2653\") "
Sep 11 17:12:45.766217 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.766161    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9999eec3-90ad-49a1-85fa-c21c230c2653-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9999eec3-90ad-49a1-85fa-c21c230c2653" (UID: "9999eec3-90ad-49a1-85fa-c21c230c2653"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:12:45.766450 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.766268    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9999eec3-90ad-49a1-85fa-c21c230c2653" (UID: "9999eec3-90ad-49a1-85fa-c21c230c2653"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:12:45.766450 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.766264    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9999eec3-90ad-49a1-85fa-c21c230c2653" (UID: "9999eec3-90ad-49a1-85fa-c21c230c2653"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:12:45.768194 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.768164    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9999eec3-90ad-49a1-85fa-c21c230c2653-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9999eec3-90ad-49a1-85fa-c21c230c2653" (UID: "9999eec3-90ad-49a1-85fa-c21c230c2653"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:12:45.768307 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.768205    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-api-access-zdkz2" (OuterVolumeSpecName: "kube-api-access-zdkz2") pod "9999eec3-90ad-49a1-85fa-c21c230c2653" (UID: "9999eec3-90ad-49a1-85fa-c21c230c2653"). InnerVolumeSpecName "kube-api-access-zdkz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:12:45.801630 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.801592    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Sep 11 17:12:45.806981 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.806960    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 11 17:12:45.867573 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.867538    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:45.867573 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.867568    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9999eec3-90ad-49a1-85fa-c21c230c2653-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:45.867573 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.867578    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9999eec3-90ad-49a1-85fa-c21c230c2653-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:45.867795 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.867586    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9999eec3-90ad-49a1-85fa-c21c230c2653-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:45.867795 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:45.867595    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdkz2\" (UniqueName: \"kubernetes.io/projected/9999eec3-90ad-49a1-85fa-c21c230c2653-kube-api-access-zdkz2\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:46.252811 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:46.252779    2576 generic.go:356] "Generic (PLEG): container finished" podID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerID="1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd" exitCode=2
Sep 11 17:12:46.253024 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:46.252850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" event={"ID":"01b8a661-54db-4aa6-a232-315cdddeffa9","Type":"ContainerDied","Data":"1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd"}
Sep 11 17:12:46.254206 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:46.254187    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_9999eec3-90ad-49a1-85fa-c21c230c2653/storage-initializer/1.log"
Sep 11 17:12:46.254320 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:46.254250    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9" event={"ID":"9999eec3-90ad-49a1-85fa-c21c230c2653","Type":"ContainerDied","Data":"d685aeb0c362be173653677a335c484acdf622cdbc970daa33f0ec949c5360e2"}
Sep 11 17:12:46.254320 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:46.254284    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"
Sep 11 17:12:46.254320 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:46.254297    2576 scope.go:117] "RemoveContainer" containerID="6bd11d94005271442346296f4cfb70fc3c7b6c39b3844cfab66d77b890253a89"
Sep 11 17:12:46.294957 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:46.294898    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"]
Sep 11 17:12:46.297329 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:46.297305    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9"]
Sep 11 17:12:46.603141 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:46.603069    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9999eec3-90ad-49a1-85fa-c21c230c2653" path="/var/lib/kubelet/pods/9999eec3-90ad-49a1-85fa-c21c230c2653/volumes"
Sep 11 17:12:50.050595 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.050569    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:12:50.094220 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.094188    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b8a661-54db-4aa6-a232-315cdddeffa9-kserve-provision-location\") pod \"01b8a661-54db-4aa6-a232-315cdddeffa9\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") "
Sep 11 17:12:50.094387 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.094225    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b8a661-54db-4aa6-a232-315cdddeffa9-proxy-tls\") pod \"01b8a661-54db-4aa6-a232-315cdddeffa9\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") "
Sep 11 17:12:50.094387 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.094260    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-cabundle-cert\") pod \"01b8a661-54db-4aa6-a232-315cdddeffa9\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") "
Sep 11 17:12:50.094387 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.094278    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-rbac-proxy-sar-config\") pod \"01b8a661-54db-4aa6-a232-315cdddeffa9\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") "
Sep 11 17:12:50.094387 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.094299    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8gcw\" (UniqueName: \"kubernetes.io/projected/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-api-access-q8gcw\") pod \"01b8a661-54db-4aa6-a232-315cdddeffa9\" (UID: \"01b8a661-54db-4aa6-a232-315cdddeffa9\") "
Sep 11 17:12:50.094600 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.094561    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b8a661-54db-4aa6-a232-315cdddeffa9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01b8a661-54db-4aa6-a232-315cdddeffa9" (UID: "01b8a661-54db-4aa6-a232-315cdddeffa9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:12:50.094659 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.094634    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01b8a661-54db-4aa6-a232-315cdddeffa9" (UID: "01b8a661-54db-4aa6-a232-315cdddeffa9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:12:50.094713 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.094663    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01b8a661-54db-4aa6-a232-315cdddeffa9" (UID: "01b8a661-54db-4aa6-a232-315cdddeffa9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:12:50.096474 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.096451    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b8a661-54db-4aa6-a232-315cdddeffa9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01b8a661-54db-4aa6-a232-315cdddeffa9" (UID: "01b8a661-54db-4aa6-a232-315cdddeffa9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:12:50.096571 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.096552    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-api-access-q8gcw" (OuterVolumeSpecName: "kube-api-access-q8gcw") pod "01b8a661-54db-4aa6-a232-315cdddeffa9" (UID: "01b8a661-54db-4aa6-a232-315cdddeffa9"). InnerVolumeSpecName "kube-api-access-q8gcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:12:50.195149 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.195117    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b8a661-54db-4aa6-a232-315cdddeffa9-kserve-provision-location\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:50.195149 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.195148    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b8a661-54db-4aa6-a232-315cdddeffa9-proxy-tls\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:50.195338 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.195157    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-cabundle-cert\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:50.195338 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.195165    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:50.195338 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.195173    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gcw\" (UniqueName: \"kubernetes.io/projected/01b8a661-54db-4aa6-a232-315cdddeffa9-kube-api-access-q8gcw\") on node \"ip-10-0-143-197.ec2.internal\" DevicePath \"\""
Sep 11 17:12:50.271012 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.270976    2576 generic.go:356] "Generic (PLEG): container finished" podID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerID="3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947" exitCode=0
Sep 11 17:12:50.271191 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.271039    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"
Sep 11 17:12:50.271191 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.271074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" event={"ID":"01b8a661-54db-4aa6-a232-315cdddeffa9","Type":"ContainerDied","Data":"3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947"}
Sep 11 17:12:50.271191 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.271139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw" event={"ID":"01b8a661-54db-4aa6-a232-315cdddeffa9","Type":"ContainerDied","Data":"eb7b4bb65b2f3b0dce9faeb842f793c5b870319123db6cb62ad49dd9bab1c7c3"}
Sep 11 17:12:50.271191 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.271166    2576 scope.go:117] "RemoveContainer" containerID="1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd"
Sep 11 17:12:50.279915 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.279897    2576 scope.go:117] "RemoveContainer" containerID="3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947"
Sep 11 17:12:50.287008 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.286991    2576 scope.go:117] "RemoveContainer" containerID="8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218"
Sep 11 17:12:50.294233 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.294216    2576 scope.go:117] "RemoveContainer" containerID="1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd"
Sep 11 17:12:50.294504 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:12:50.294482    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd\": container with ID starting with 1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd not found: ID does not exist" containerID="1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd"
Sep 11 17:12:50.294583 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.294503    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"]
Sep 11 17:12:50.294583 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.294513    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd"} err="failed to get container status \"1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd\": rpc error: code = NotFound desc = could not find container \"1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd\": container with ID starting with 1b38d1e22ebda3d361a2521643e5a2e5523e762fc6cf59d019e54ce075f888fd not found: ID does not exist"
Sep 11 17:12:50.294583 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.294530    2576 scope.go:117] "RemoveContainer" containerID="3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947"
Sep 11 17:12:50.294796 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:12:50.294778    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947\": container with ID starting with 3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947 not found: ID does not exist" containerID="3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947"
Sep 11 17:12:50.294841 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.294802    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947"} err="failed to get container status \"3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947\": rpc error: code = NotFound desc = could not find container \"3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947\": container with ID starting with 3284fa10278136b67fdd462cb34facbccfc10bf4d285eb8cbd03f16e6a988947 not found: ID does not exist"
Sep 11 17:12:50.294841 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.294818    2576 scope.go:117] "RemoveContainer" containerID="8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218"
Sep 11 17:12:50.295074 ip-10-0-143-197 kubenswrapper[2576]: E0911 17:12:50.295057    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218\": container with ID starting with 8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218 not found: ID does not exist" containerID="8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218"
Sep 11 17:12:50.295129 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.295079    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218"} err="failed to get container status \"8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218\": rpc error: code = NotFound desc = could not find container \"8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218\": container with ID starting with 8db8974d46314264eb6cc3022a9ba15040d2b1ef5d8d196232eec7738d135218 not found: ID does not exist"
Sep 11 17:12:50.300653 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.300632    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw"]
Sep 11 17:12:50.602840 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:50.602760    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" path="/var/lib/kubelet/pods/01b8a661-54db-4aa6-a232-315cdddeffa9/volumes"
Sep 11 17:12:51.188685 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:51.188651    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:51.282719 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:51.282686    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:59.365587 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:12:59.365551    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:11.023331 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:11.023296    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:27.930657 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:27.930623    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:28.052127 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052097    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w5hdz/must-gather-8cc59"]
Sep 11 17:13:28.052370 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052359    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9999eec3-90ad-49a1-85fa-c21c230c2653" containerName="storage-initializer"
Sep 11 17:13:28.052415 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052372    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9999eec3-90ad-49a1-85fa-c21c230c2653" containerName="storage-initializer"
Sep 11 17:13:28.052415 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052383    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9999eec3-90ad-49a1-85fa-c21c230c2653" containerName="storage-initializer"
Sep 11 17:13:28.052415 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052389    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9999eec3-90ad-49a1-85fa-c21c230c2653" containerName="storage-initializer"
Sep 11 17:13:28.052415 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052398    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kube-rbac-proxy"
Sep 11 17:13:28.052415 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052403    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kube-rbac-proxy"
Sep 11 17:13:28.052415 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052414    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="storage-initializer"
Sep 11 17:13:28.052598 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052419    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="storage-initializer"
Sep 11 17:13:28.052598 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052430    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container"
Sep 11 17:13:28.052598 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052435    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container"
Sep 11 17:13:28.052598 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052478    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9999eec3-90ad-49a1-85fa-c21c230c2653" containerName="storage-initializer"
Sep 11 17:13:28.052598 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052484    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kube-rbac-proxy"
Sep 11 17:13:28.052598 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052492    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b8a661-54db-4aa6-a232-315cdddeffa9" containerName="kserve-container"
Sep 11 17:13:28.052598 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.052497    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9999eec3-90ad-49a1-85fa-c21c230c2653" containerName="storage-initializer"
Sep 11 17:13:28.055341 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.055316    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5hdz/must-gather-8cc59"
Sep 11 17:13:28.057997 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.057978    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w5hdz\"/\"default-dockercfg-4gl5p\""
Sep 11 17:13:28.058117 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.058033    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5hdz\"/\"kube-root-ca.crt\""
Sep 11 17:13:28.058302 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.058286    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5hdz\"/\"openshift-service-ca.crt\""
Sep 11 17:13:28.067383 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.067361    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5hdz/must-gather-8cc59"]
Sep 11 17:13:28.070692 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.070666    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:28.169734 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.169704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr2n4\" (UniqueName: \"kubernetes.io/projected/b88c28fa-181a-4646-b2db-ef59f0a0f57f-kube-api-access-cr2n4\") pod \"must-gather-8cc59\" (UID: \"b88c28fa-181a-4646-b2db-ef59f0a0f57f\") " pod="openshift-must-gather-w5hdz/must-gather-8cc59"
Sep 11 17:13:28.169911 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.169779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b88c28fa-181a-4646-b2db-ef59f0a0f57f-must-gather-output\") pod \"must-gather-8cc59\" (UID: \"b88c28fa-181a-4646-b2db-ef59f0a0f57f\") " pod="openshift-must-gather-w5hdz/must-gather-8cc59"
Sep 11 17:13:28.169911 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.169778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:28.270592 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.270554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2n4\" (UniqueName: \"kubernetes.io/projected/b88c28fa-181a-4646-b2db-ef59f0a0f57f-kube-api-access-cr2n4\") pod \"must-gather-8cc59\" (UID: \"b88c28fa-181a-4646-b2db-ef59f0a0f57f\") " pod="openshift-must-gather-w5hdz/must-gather-8cc59"
Sep 11 17:13:28.270812 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.270645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b88c28fa-181a-4646-b2db-ef59f0a0f57f-must-gather-output\") pod \"must-gather-8cc59\" (UID: \"b88c28fa-181a-4646-b2db-ef59f0a0f57f\") " pod="openshift-must-gather-w5hdz/must-gather-8cc59"
Sep 11 17:13:28.271049 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.271028    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b88c28fa-181a-4646-b2db-ef59f0a0f57f-must-gather-output\") pod \"must-gather-8cc59\" (UID: \"b88c28fa-181a-4646-b2db-ef59f0a0f57f\") " pod="openshift-must-gather-w5hdz/must-gather-8cc59"
Sep 11 17:13:28.280340 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.280318    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2n4\" (UniqueName: \"kubernetes.io/projected/b88c28fa-181a-4646-b2db-ef59f0a0f57f-kube-api-access-cr2n4\") pod \"must-gather-8cc59\" (UID: \"b88c28fa-181a-4646-b2db-ef59f0a0f57f\") " pod="openshift-must-gather-w5hdz/must-gather-8cc59"
Sep 11 17:13:28.363645 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.363615    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5hdz/must-gather-8cc59"
Sep 11 17:13:28.636154 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.636093    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5hdz/must-gather-8cc59"]
Sep 11 17:13:28.638830 ip-10-0-143-197 kubenswrapper[2576]: W0911 17:13:28.638802    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb88c28fa_181a_4646_b2db_ef59f0a0f57f.slice/crio-eb1c281f3ac0d13240ac55c5c5d0f104ee728da1bbb1f09f0a6e257454d8f02c WatchSource:0}: Error finding container eb1c281f3ac0d13240ac55c5c5d0f104ee728da1bbb1f09f0a6e257454d8f02c: Status 404 returned error can't find the container with id eb1c281f3ac0d13240ac55c5c5d0f104ee728da1bbb1f09f0a6e257454d8f02c
Sep 11 17:13:28.641113 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:28.641097    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:13:29.416259 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:29.416217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdz/must-gather-8cc59" event={"ID":"b88c28fa-181a-4646-b2db-ef59f0a0f57f","Type":"ContainerStarted","Data":"eb1c281f3ac0d13240ac55c5c5d0f104ee728da1bbb1f09f0a6e257454d8f02c"}
Sep 11 17:13:30.423848 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:30.423761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdz/must-gather-8cc59" event={"ID":"b88c28fa-181a-4646-b2db-ef59f0a0f57f","Type":"ContainerStarted","Data":"0d5ebc2d7589f3d5e1b1f783888dc7be87f859f3538a49a2418b7f97f59a8187"}
Sep 11 17:13:30.423848 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:30.423816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdz/must-gather-8cc59" event={"ID":"b88c28fa-181a-4646-b2db-ef59f0a0f57f","Type":"ContainerStarted","Data":"e027be8fcfe89e50322ac8265a5bcec96dba1a1bd6f4e07ab1ca564933567d5c"}
Sep 11 17:13:30.444488 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:30.444435    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w5hdz/must-gather-8cc59" podStartSLOduration=1.539761621 podStartE2EDuration="2.444417626s" totalImagesPullingTime="904.656005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:13:30.442767481 +0000 UTC m=+3404.352568642" watchObservedRunningTime="2026-09-11 17:13:30.444417626 +0000 UTC m=+3404.354218832"
Sep 11 17:13:31.198976 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:31.198928    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4gcmq_74c8f10b-e859-4c3e-b95a-5f58ee099241/global-pull-secret-syncer/0.log"
Sep 11 17:13:31.353187 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:31.353150    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log"
Sep 11 17:13:31.498407 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:31.498315    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-197.ec2.internal_83c5fb3c2cd6a94582ba9e97cbdde139/haproxy/0.log"
Sep 11 17:13:34.198651 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:34.198616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:34.299087 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:34.299020    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-48rfb_37d25eeb-883f-4742-a291-61e8a28f44fe/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:36.421569 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:36.421511    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pgfps_65da7519-5eab-41d8-9009-076d71e5c154/node-exporter/0.log"
Sep 11 17:13:36.447286 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:36.447253    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pgfps_65da7519-5eab-41d8-9009-076d71e5c154/kube-rbac-proxy/0.log"
Sep 11 17:13:36.478081 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:36.478056    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pgfps_65da7519-5eab-41d8-9009-076d71e5c154/init-textfile/0.log"
Sep 11 17:13:39.066200 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.066170    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/2.log"
Sep 11 17:13:39.072692 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.072670    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6jh9m_0f563067-de11-4c7f-92eb-0cdefa727b88/console-operator/3.log"
Sep 11 17:13:39.157806 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.156377    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"]
Sep 11 17:13:39.161700 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.161666    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.168042 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.168017    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"]
Sep 11 17:13:39.262692 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.262659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zldq\" (UniqueName: \"kubernetes.io/projected/ad878702-4cce-4d12-9387-7328a5f6a180-kube-api-access-4zldq\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.262907 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.262714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-lib-modules\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.262907 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.262740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-proc\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.262907 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.262817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-sys\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.262907 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.262888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-podres\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.363617 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.363527    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zldq\" (UniqueName: \"kubernetes.io/projected/ad878702-4cce-4d12-9387-7328a5f6a180-kube-api-access-4zldq\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.363617 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.363614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-lib-modules\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.363868 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.363650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-proc\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.363868 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.363694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-sys\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.363868 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.363745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-podres\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.364069 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.363918    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-podres\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.364400 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.364369    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-lib-modules\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.364516 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.364440    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-proc\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.364516 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.364481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad878702-4cce-4d12-9387-7328a5f6a180-sys\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.376816 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.376783    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zldq\" (UniqueName: \"kubernetes.io/projected/ad878702-4cce-4d12-9387-7328a5f6a180-kube-api-access-4zldq\") pod \"perf-node-gather-daemonset-ddnvb\" (UID: \"ad878702-4cce-4d12-9387-7328a5f6a180\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.474689 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.474655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:39.750598 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:39.750554    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"]
Sep 11 17:13:40.490871 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:40.490828    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb" event={"ID":"ad878702-4cce-4d12-9387-7328a5f6a180","Type":"ContainerStarted","Data":"05f15055161a74595351c7b95999c1de33258dd6b9b2ca4d5d9ad0d1d26e435d"}
Sep 11 17:13:40.491364 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:40.490900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb" event={"ID":"ad878702-4cce-4d12-9387-7328a5f6a180","Type":"ContainerStarted","Data":"dd85f2deabee5d31d78e4867eae2dad2020619e57e178b02e21984fd453eb0eb"}
Sep 11 17:13:40.491744 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:40.491716    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:41.098135 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:41.098109    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzth4_b250f482-a270-4c37-89a0-69fee0abc0ae/dns/0.log"
Sep 11 17:13:41.121178 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:41.121151    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rzth4_b250f482-a270-4c37-89a0-69fee0abc0ae/kube-rbac-proxy/0.log"
Sep 11 17:13:41.148532 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:41.148491    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8bwsp_db333929-74ef-449c-a79a-6878ef6051c1/dns-node-resolver/0.log"
Sep 11 17:13:41.770622 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:41.770594    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sjw2m_5e2a6297-0738-43c3-9d88-7cfa8d9474d7/node-ca/0.log"
Sep 11 17:13:42.953449 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:42.953415    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dttkg_52a2aecb-8c51-4bbc-80ff-7ccc0d0c769e/serve-healthcheck-canary/0.log"
Sep 11 17:13:43.377571 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:43.377490    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bbldx_1a163692-fd52-4032-8ae6-3966412e2561/kube-rbac-proxy/0.log"
Sep 11 17:13:43.402473 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:43.402437    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bbldx_1a163692-fd52-4032-8ae6-3966412e2561/exporter/0.log"
Sep 11 17:13:43.426719 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:43.426690    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bbldx_1a163692-fd52-4032-8ae6-3966412e2561/extractor/0.log"
Sep 11 17:13:45.924852 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:45.924822    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-656b75fc8d-lkvx9_d98ac13e-000e-4a57-8f4b-ab46bbd50895/manager/0.log"
Sep 11 17:13:45.981809 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:45.981777    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-jdqhw_59abc94d-ea3c-4e55-9faf-ccba8939ec71/server/0.log"
Sep 11 17:13:46.313548 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:46.313515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-nl2xp_5751c273-e764-4438-b5f9-9975a0e07e3b/manager/0.log"
Sep 11 17:13:46.495799 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:46.495771    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-wc8vk_3a626c9e-ba00-4e72-8cf9-ed7b0f857a5d/seaweedfs/0.log"
Sep 11 17:13:46.524427 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:46.524402    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-mcd7s_c881b739-c8c7-474e-94a6-3e436b6aa044/seaweedfs-tls-custom/0.log"
Sep 11 17:13:47.514389 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:47.514362    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb"
Sep 11 17:13:47.533906 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:47.533854    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-ddnvb" podStartSLOduration=8.533838977 podStartE2EDuration="8.533838977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:13:40.51057816 +0000 UTC m=+3414.420379324" watchObservedRunningTime="2026-09-11 17:13:47.533838977 +0000 UTC m=+3421.443640139"
Sep 11 17:13:50.996794 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:50.996768    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-wdxwv_6eb31607-be58-459e-b1bf-a1b96c2e68d4/migrator/0.log"
Sep 11 17:13:51.023189 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:51.023150    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-wdxwv_6eb31607-be58-459e-b1bf-a1b96c2e68d4/graceful-termination/0.log"
Sep 11 17:13:52.560733 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:52.560707    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7w7vl_a9648594-e0ca-46fd-ab95-86cfaa507bae/kube-multus-additional-cni-plugins/0.log"
Sep 11 17:13:52.604053 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:52.604030    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7w7vl_a9648594-e0ca-46fd-ab95-86cfaa507bae/egress-router-binary-copy/0.log"
Sep 11 17:13:52.642606 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:52.642580    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7w7vl_a9648594-e0ca-46fd-ab95-86cfaa507bae/cni-plugins/0.log"
Sep 11 17:13:52.675991 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:52.675963    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7w7vl_a9648594-e0ca-46fd-ab95-86cfaa507bae/bond-cni-plugin/0.log"
Sep 11 17:13:52.707443 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:52.707415    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7w7vl_a9648594-e0ca-46fd-ab95-86cfaa507bae/routeoverride-cni/0.log"
Sep 11 17:13:52.744455 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:52.744428    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7w7vl_a9648594-e0ca-46fd-ab95-86cfaa507bae/whereabouts-cni-bincopy/0.log"
Sep 11 17:13:52.775047 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:52.775022    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7w7vl_a9648594-e0ca-46fd-ab95-86cfaa507bae/whereabouts-cni/0.log"
Sep 11 17:13:53.208013 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:53.207982    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mn7vw_ea980b3b-40fd-4ea0-a4b3-70706ab8c942/kube-multus/0.log"
Sep 11 17:13:53.232010 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:53.231987    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9sg48_e45bda4f-12ee-46a5-8d83-7a1602904679/network-metrics-daemon/0.log"
Sep 11 17:13:53.259878 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:53.259850    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9sg48_e45bda4f-12ee-46a5-8d83-7a1602904679/kube-rbac-proxy/0.log"
Sep 11 17:13:54.205104 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:54.205065    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-controller/0.log"
Sep 11 17:13:54.225032 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:54.225009    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/0.log"
Sep 11 17:13:54.225187 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:54.225050    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovn-acl-logging/1.log"
Sep 11 17:13:54.245755 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:54.245732    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/kube-rbac-proxy-node/0.log"
Sep 11 17:13:54.269664 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:54.269634    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 17:13:54.292993 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:54.292930    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/northd/0.log"
Sep 11 17:13:54.320297 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:54.320276    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/nbdb/0.log"
Sep 11 17:13:54.344508 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:54.344481    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/sbdb/0.log"
Sep 11 17:13:54.397279 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:54.397227    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gfkd5_c911c447-3778-4e0c-bd78-297c361a8740/ovnkube-controller/0.log"
Sep 11 17:13:55.931519 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:55.931495    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-hwvmz_61eeb2f0-b6c0-4286-80b8-3fc3268a8d90/check-endpoints/0.log"
Sep 11 17:13:55.986913 ip-10-0-143-197 kubenswrapper[2576]: I0911 17:13:55.986883    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dd2bv_df08f2ff-be81-4cc3-966d-bc1be8960c4a/network-check-target-container/0.log"