Sep 17 19:06:19.297912 ip-10-0-140-110 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 19:06:22.800371 ip-10-0-140-110 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:22.800371 ip-10-0-140-110 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 19:06:22.800371 ip-10-0-140-110 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:22.800371 ip-10-0-140-110 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 19:06:22.800371 ip-10-0-140-110 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:22.801082 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.800446    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 19:06:22.805480 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805464    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:22.805480 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805479    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805483    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805487    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805490    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805493    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805498    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805501    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805505    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805508    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805511    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805514    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805516    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805519    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805522    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805530    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805532    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805535    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805538    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805540    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:22.805549 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805543    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805547    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805549    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805552    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805555    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805558    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805560    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805563    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805566    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805568    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805571    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805574    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805576    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805579    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805581    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805584    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805586    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805589    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805591    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805594    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:22.806025 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805596    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805598    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805601    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805603    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805606    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805609    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805626    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805629    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805632    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805634    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805637    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805664    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805667    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805670    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805675    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805678    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805681    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805684    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805686    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805689    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:22.806547 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805691    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805694    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805697    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805699    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805702    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805704    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805708    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805712    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805716    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805720    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805724    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805727    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805730    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805732    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805735    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805738    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805743    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805745    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805748    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:22.807051 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805751    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805753    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805756    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805759    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805761    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805764    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805766    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805770    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805773    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805775    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805778    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805781    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805783    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805786    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805789    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805791    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805794    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805796    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805799    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.805801    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:22.807518 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806486    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806492    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806495    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806498    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806501    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806504    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806506    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806509    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806511    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806514    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806517    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806520    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806523    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806525    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806528    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806531    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806533    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806535    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806538    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:22.808018 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806543    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806547    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806550    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806553    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806556    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806558    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806561    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806564    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806566    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806569    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806571    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806574    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806577    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806580    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806582    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806585    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806587    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806589    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806592    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806594    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:22.808508 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806597    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806599    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806601    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806604    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806607    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806610    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806628    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806631    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806634    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806636    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806639    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806642    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806644    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806647    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806650    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806652    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806655    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806657    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806660    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806662    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:22.809077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806665    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806669    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806673    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806677    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806679    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806682    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806684    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806687    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806689    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806692    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806694    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806697    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806700    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806703    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806705    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806708    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806710    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806713    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806716    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806719    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:22.809581 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806722    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806724    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806727    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806730    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806735    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806738    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806741    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806744    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806747    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806749    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806752    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806755    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806757    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806760    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806762    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806765    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806767    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806769    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806772    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:22.810083 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.806774    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807135    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807140    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807143    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807145    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807148    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807152    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807156    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807159    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807162    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807166    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807169    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807172    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807175    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807178    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807181    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807184    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807187    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807190    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807192    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:22.810592 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807195    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807197    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807200    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807203    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807206    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807208    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807210    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807213    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807216    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807219    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807221    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807223    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807226    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807228    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807231    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807233    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807236    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807238    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807241    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807243    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:22.811160 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807246    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807248    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807251    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807253    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807256    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807259    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807261    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807264    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807266    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807269    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807272    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807274    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807277    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807279    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807281    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807284    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807287    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807290    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807292    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:22.811788 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807294    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807297    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807299    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807302    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807304    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807307    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807309    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807313    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807315    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807318    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807320    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807323    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807325    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807327    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807330    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807333    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807335    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807338    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807341    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:22.812293 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807343    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807347    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807350    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807354    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807357    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807360    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807362    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807365    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807368    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807370    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807375    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807378    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807380    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807383    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807385    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807388    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807391    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807393    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807396    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:22.812786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807398    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807401    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.807404    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807475    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807507    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807512    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807517    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807521    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807524    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807528    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807532    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807538    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807542    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807545    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807548    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807554    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807556    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807559    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807562    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807565    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807568    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807573    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807576    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807579    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 19:06:22.813246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807582    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807585    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807589    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807592    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807595    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807598    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807601    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807604    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807607    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807610    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807627    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807631    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807634    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807637    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807640    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807643    2583 flags.go:64] FLAG: --enable-server="true"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807646    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807650    2583 flags.go:64] FLAG: --event-burst="100"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807653    2583 flags.go:64] FLAG: --event-qps="50"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807657    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807661    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807664    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807668    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807672    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807675    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 19:06:22.813828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807678    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807681    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807684    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807687    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807689    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807692    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807695    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807698    2583 flags.go:64] FLAG: --feature-gates=""
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807701    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807704    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807708    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807711    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807713    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807716    2583 flags.go:64] FLAG: --help="false"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807719    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-140-110.ec2.internal"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807722    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807725    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807727    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807731    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807734    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807736    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807739    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807742    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 19:06:22.814440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807744    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807747    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807750    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807756    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807759    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807761    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807765    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807769    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807772    2583 flags.go:64] FLAG: --lock-file=""
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807774    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807777    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807780    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807786    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807788    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807791    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807794    2583 flags.go:64] FLAG: --logging-format="text"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807796    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807799    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807802    2583 flags.go:64] FLAG: --manifest-url=""
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807805    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807809    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807812    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807816    2583 flags.go:64] FLAG: --max-pods="110"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807819    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807822    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 19:06:22.815092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807825    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807828    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807830    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807833    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807836    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807843    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807845    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807848    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807851    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807854    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807859    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807863    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807866    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807869    2583 flags.go:64] FLAG: --port="10250"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807874    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807877    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08159ea3d09073dc1"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807880    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807883    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807886    2583 flags.go:64] FLAG: --register-node="true"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807889    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807892    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807896    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807898    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807901    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807904    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 19:06:22.815762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807907    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807910    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807913    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807916    2583 flags.go:64] FLAG: --runonce="false"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807918    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807921    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807924    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807927    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807930    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807932    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807935    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807938    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807941    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807944    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807946    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807949    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807952    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807955    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807959    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807964    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807966    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807970    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807975    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807981    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807984    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 19:06:22.816665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807987    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807990    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807993    2583 flags.go:64] FLAG: --v="2"
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.807996    2583 flags.go:64] FLAG: --version="false"
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.808000    2583 flags.go:64] FLAG: --vmodule=""
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.808003    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.808007    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808119    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808124    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808127    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808130    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808133    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808135    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808138    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808141    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808144    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808146    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808149    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808151    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808154    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808157    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:22.817578 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808159    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808162    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808164    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808166    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808172    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808175    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808177    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808181    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808184    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808187    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808189    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808192    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808194    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808197    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808200    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808202    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808205    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808207    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808210    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:22.818157 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808213    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808215    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808218    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808220    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808223    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808226    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808228    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808231    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808234    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808236    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808239    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808241    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808244    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808247    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808251    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808253    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808256    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808260    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808263    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:22.818640 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808265    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808268    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808271    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808273    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808276    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808280    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808284    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808287    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808289    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808292    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808295    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808298    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808301    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808304    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808307    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808310    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808312    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808315    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808318    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808320    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:22.819109 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808323    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808325    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808328    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808331    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808333    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808336    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808338    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808341    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808343    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808346    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808350    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808353    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808355    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808358    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808360    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808362    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808365    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808368    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808371    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808373    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:22.819662 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808376    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808378    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808381    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808384    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808386    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808390    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.808393    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.808959    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.817642    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.817657    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.817677    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.817686    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817740    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817745    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:22.820158 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817748    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817751    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817754    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817757    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817760    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817763    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817765    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817768    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817773    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817777    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817781    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817784    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817787    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817790    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817792    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817796    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817798    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817801    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817804    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:22.820504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817807    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817817    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817820    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817823    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817826    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817830    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817834    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817837    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817841    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817845    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817848    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817850    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817853    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817856    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817858    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817861    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817864    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817866    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817869    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817871    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:22.821028 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817874    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817877    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817879    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817882    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817884    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817887    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817890    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817892    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817895    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817897    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817900    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817902    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817905    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817907    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817910    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817912    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817915    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817917    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817920    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817922    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:22.821522 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817926    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817929    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817931    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817934    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817936    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817939    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817942    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817944    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817947    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817949    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817952    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817955    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817957    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817960    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817963    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817965    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817968    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817970    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817974    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817976    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:22.822020 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817979    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817982    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817984    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817987    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817990    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817992    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817995    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.817998    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818000    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818003    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818005    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818008    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818011    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818014    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818016    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818019    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818022    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:22.822512 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818024    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.818029    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818150    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818156    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818159    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818162    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818165    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818168    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818171    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818175    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818178    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818182    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818184    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818187    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818190    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:22.822946 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818193    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818195    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818198    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818200    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818203    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818205    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818208    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818211    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818213    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818216    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818218    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818220    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818223    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818226    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818229    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818231    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818234    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818236    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818239    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818241    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:22.823324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818244    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818246    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818249    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818251    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818254    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818256    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818259    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818261    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818264    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818266    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818269    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818271    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818274    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818277    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818279    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818281    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818285    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818289    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818292    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818295    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:22.823817 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818297    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818300    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818303    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818305    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818309    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818312    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818314    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818317    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818320    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818323    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818326    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818328    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818331    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818334    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818338    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818340    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818343    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818346    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818348    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:22.824295 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818351    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818353    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818356    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818358    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818361    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818363    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818366    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818369    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818371    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818374    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818376    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818379    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818381    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818384    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818386    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818389    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818391    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818395    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818397    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818400    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:22.824811 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818403    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818405    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818407    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818410    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818412    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818415    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:22.818417    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.818422    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.818524    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.820628    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.822177    2583 server.go:1019] "Starting client certificate rotation"
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.822275    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:22.825285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.823157    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:22.825897 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.825884    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:22.827079 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.827062    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:22.839643 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.839626    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 17 19:06:22.841721 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.841647    2583 log.go:25] "Validated CRI v1 image API"
Sep 17 19:06:22.842479 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.842465    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 19:06:22.844828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.844810    2583 fs.go:135] Filesystem UUIDs: map[0d9053c9-8907-4c1c-8baf-be3a78c21428:/dev/nvme0n1p3 614290e0-19b1-4c44-9792-6a73ee2ad696:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 19:06:22.844917 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.844827    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 19:06:22.849429 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.849301    2583 manager.go:216] Machine: {Timestamp:2026-09-17 19:06:22.848830722 +0000 UTC m=+1.976572113 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096176 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec255995dd79a5fe45b9137f9548bb3d SystemUUID:ec255995-dd79-a5fe-45b9-137f9548bb3d BootID:b4dd2b21-8834-4e5b-bf52-8047e4f92aa3 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:b4:50:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:b4:50:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:82:62:e8:f6:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 19:06:22.849429 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.849419    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 19:06:22.849595 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.849492    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 19:06:22.851273 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.851244    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 19:06:22.851406 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.851277    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 19:06:22.852472 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.852460    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 19:06:22.852506 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.852475    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 19:06:22.852506 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.852483    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:22.852506 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.852493    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:22.852594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.852556    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 19:06:22.853702 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.853687    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:22.855436 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.855421    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:22.855879 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.855862    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 19:06:22.856039 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.856015    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 19:06:22.856151 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.856044    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 19:06:22.856151 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.856060    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 19:06:22.856151 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.856076    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 17 19:06:22.856151 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.856090    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 19:06:22.856506 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.856492    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:22.856561 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.856517    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:22.860354 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.860335    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 19:06:22.862565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.862552    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 19:06:22.862607 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.862583    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 19:06:22.863838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863827    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 19:06:22.863870 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863845    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 19:06:22.863870 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863851    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 19:06:22.863870 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863859    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863872    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863880    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863893    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863898    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863905    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863910    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863918    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863923    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863942    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 19:06:22.863948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.863950    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 19:06:22.867790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.867773    2583 server.go:1268] "Started kubelet"
Sep 17 19:06:22.867879 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.867837    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 19:06:22.867926 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.867846    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 19:06:22.867957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.867941    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 19:06:22.868698 ip-10-0-140-110 systemd[1]: Started Kubernetes Kubelet.
Sep 17 19:06:22.869710 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.869029    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 19:06:22.869710 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.869137    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 19:06:22.871323 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.871303    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 19:06:22.871917 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.871898    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 19:06:22.871977 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.871898    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 19:06:22.873390 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.873365    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 19:06:22.874045 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.874028    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 19:06:22.874141 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.874055    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:22.874198 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.874169    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:22.874708 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.874662    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 19:06:22.874708 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.874663    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 19:06:22.874708 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.874690    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 19:06:22.874887 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.874761    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 19:06:22.874887 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.874770    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 19:06:22.876979 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.876962    2583 factory.go:55] Registering systemd factory
Sep 17 19:06:22.876979 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.876979    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 17 19:06:22.877189 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.877169    2583 factory.go:153] Registering CRI-O factory
Sep 17 19:06:22.877189 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.877190    2583 factory.go:223] Registration of the crio container factory successfully
Sep 17 19:06:22.877318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.877239    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 19:06:22.877318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.877256    2583 factory.go:103] Registering Raw factory
Sep 17 19:06:22.877318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.877272    2583 manager.go:1196] Started watching for new ooms in manager
Sep 17 19:06:22.877660 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.877647    2583 manager.go:349] Starting recovery of all containers
Sep 17 19:06:22.898991 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.898947    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 19:06:22.900788 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.900742    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 19:06:22.906287 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.905183    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-110.ec2.internal.18d6312fb34d0f4a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-110.ec2.internal,UID:ip-10-0-140-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-110.ec2.internal,},FirstTimestamp:2026-09-17 19:06:22.867746634 +0000 UTC m=+1.995488027,LastTimestamp:2026-09-17 19:06:22.867746634 +0000 UTC m=+1.995488027,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-110.ec2.internal,}"
Sep 17 19:06:22.907199 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.907187    2583 manager.go:354] Recovery completed
Sep 17 19:06:22.907290 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.907272    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fc4wk"
Sep 17 19:06:22.911208 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.911196    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:22.914125 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.914105    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:22.914217 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.914144    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:22.914217 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.914159    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:22.914756 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.914740    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 19:06:22.914756 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.914752    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 19:06:22.914889 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.914760    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:22.914889 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.914767    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:22.915440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.915425    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:22.915486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.915450    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:22.915486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.915460    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:22.915906 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.915895    2583 policy_none.go:49] "None policy: Start"
Sep 17 19:06:22.915944 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.915911    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 19:06:22.915944 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.915921    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 19:06:22.916075 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.916062    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fc4wk"
Sep 17 19:06:22.916339 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.916275    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-110.ec2.internal.18d6312fb610bde2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-110.ec2.internal,UID:ip-10-0-140-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-110.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-110.ec2.internal,},FirstTimestamp:2026-09-17 19:06:22.914125282 +0000 UTC m=+2.041866677,LastTimestamp:2026-09-17 19:06:22.914125282 +0000 UTC m=+2.041866677,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-110.ec2.internal,}"
Sep 17 19:06:22.917383 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.917371    2583 policy_none.go:47] "Start"
Sep 17 19:06:22.950286 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.950163    2583 manager.go:337] "Starting Device Plugin manager"
Sep 17 19:06:22.964995 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.950338    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 19:06:22.964995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.950356    2583 server.go:87] "Starting device plugin registration server"
Sep 17 19:06:22.964995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.950594    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 19:06:22.964995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.950630    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 19:06:22.964995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.950759    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 19:06:22.964995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.951567    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 19:06:22.964995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:22.951581    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 19:06:22.964995 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.952113    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 19:06:22.964995 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:22.952195    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.031385 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.031357    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 19:06:23.032646 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.032632    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 19:06:23.032747 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.032696    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 19:06:23.032747 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.032723    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 19:06:23.032845 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.032782    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 19:06:23.034981 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.034958    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:23.051080 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.051029    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.051916 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.051898    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.051991 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.051937    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.051991 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.051951    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.051991 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.051973    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.060177 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.060159    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.060224 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.060179    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-110.ec2.internal\": node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.082074 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.082051    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.133354 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.133329    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"]
Sep 17 19:06:23.133460 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.133397    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.134321 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.134306    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.134385 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.134340    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.134385 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.134359    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.135783 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.135771    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.135877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.135863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.135936 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.135906    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.136557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.136542    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.136645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.136572    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.136645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.136581    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.136645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.136542    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.136739 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.136647    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.136739 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.136657    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.137783 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.137770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.137831 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.137794    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:23.138464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.138448    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:23.138539 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.138476    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:23.138539 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.138504    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:23.163402 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.163384    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-110.ec2.internal\" not found" node="ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.167983 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.167959    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-110.ec2.internal\" not found" node="ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.176281 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.176263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ef18637fd00ca6d2dbdae6b5266649f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"8ef18637fd00ca6d2dbdae6b5266649f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.176368 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.176296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ef18637fd00ca6d2dbdae6b5266649f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"8ef18637fd00ca6d2dbdae6b5266649f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.176368 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.176322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4eb197abc8b394d582a58b08fa210ad1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"4eb197abc8b394d582a58b08fa210ad1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.182909 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.182890    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.277154 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.277121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ef18637fd00ca6d2dbdae6b5266649f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"8ef18637fd00ca6d2dbdae6b5266649f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.277154 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.277165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ef18637fd00ca6d2dbdae6b5266649f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"8ef18637fd00ca6d2dbdae6b5266649f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.277369 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.277227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ef18637fd00ca6d2dbdae6b5266649f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"8ef18637fd00ca6d2dbdae6b5266649f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.277369 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.277250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ef18637fd00ca6d2dbdae6b5266649f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"8ef18637fd00ca6d2dbdae6b5266649f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.277369 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.277272    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4eb197abc8b394d582a58b08fa210ad1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"4eb197abc8b394d582a58b08fa210ad1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.277369 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.277298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4eb197abc8b394d582a58b08fa210ad1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"4eb197abc8b394d582a58b08fa210ad1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.283013 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.282995    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.383696 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.383644    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.466863 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.466815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.472340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.472320    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.484581 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.484556    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.585070 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.585027    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.685504 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.685485    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.786204 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.786168    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.821339 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.821316    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 19:06:23.821723 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.821466    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:23.886994 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:23.886969    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Sep 17 19:06:23.911997 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.911972    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:23.920909 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.920874    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 19:01:22 +0000 UTC" deadline="2028-03-02 06:43:19.196789328 +0000 UTC"
Sep 17 19:06:23.920909 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.920906    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12755h36m55.275885489s"
Sep 17 19:06:23.975292 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.975226    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:23.991352 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.991334    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:23.992985 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:23.992971    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Sep 17 19:06:24.002339 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.002317    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:24.070944 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.070918    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:24.073063 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.073047    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:24.114570 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:24.114524    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef18637fd00ca6d2dbdae6b5266649f.slice/crio-5c027271a78dbe7442bba6e729dc8b2b483a696a946e7b99041c395bf21ba169 WatchSource:0}: Error finding container 5c027271a78dbe7442bba6e729dc8b2b483a696a946e7b99041c395bf21ba169: Status 404 returned error can't find the container with id 5c027271a78dbe7442bba6e729dc8b2b483a696a946e7b99041c395bf21ba169
Sep 17 19:06:24.119394 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.119379    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:06:24.436996 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:24.436963    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb197abc8b394d582a58b08fa210ad1.slice/crio-3ea24a090456deb909db972347ae83524c091a364895ed75b2d3504176788cb6 WatchSource:0}: Error finding container 3ea24a090456deb909db972347ae83524c091a364895ed75b2d3504176788cb6: Status 404 returned error can't find the container with id 3ea24a090456deb909db972347ae83524c091a364895ed75b2d3504176788cb6
Sep 17 19:06:24.858039 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.858007    2583 apiserver.go:52] "Watching apiserver"
Sep 17 19:06:24.866052 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.866028    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 19:06:24.866446 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.866420    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm","openshift-cluster-node-tuning-operator/tuned-p6pfc","openshift-dns/node-resolver-jqm7n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal","openshift-multus/multus-7mb94","openshift-multus/multus-additional-cni-plugins-vp9lm","openshift-network-operator/iptables-alerter-g2dhw","kube-system/global-pull-secret-syncer-xhl95","kube-system/konnectivity-agent-s7wdr","openshift-image-registry/node-ca-8fmwv","openshift-multus/network-metrics-daemon-g886c","openshift-network-diagnostics/network-check-target-vdcz8","openshift-ovn-kubernetes/ovnkube-node-jtf4n","kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"]
Sep 17 19:06:24.868033 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.868007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.869550 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.869525    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.870633 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.870589    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 19:06:24.870718 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.870688    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.870718 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.870698    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.870718 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.870707    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kfxr6\""
Sep 17 19:06:24.871536 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.871521    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gh6hc\""
Sep 17 19:06:24.871647 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.871611    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.871852 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.871835    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:24.872594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.872566    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.872939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.872922    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.876355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.874175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9sd8p\""
Sep 17 19:06:24.876355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.874300    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.876355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.874365    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.876355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.874593    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:24.876355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.874852    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.876355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.875359    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.876355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.875604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8rk6m\""
Sep 17 19:06:24.876355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.875839    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 19:06:24.876355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.876167    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 19:06:24.877905 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.877849    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:24.879295 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.879277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 19:06:24.879521 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.879501    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 19:06:24.879634 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.879530    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ckq7n\""
Sep 17 19:06:24.879928 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.879913    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.880427 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.880409    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:24.880516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.880486    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 19:06:24.880568 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:24.880514    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:24.881828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.881811    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:24.883048 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.883009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:24.884527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.884510    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:24.884640 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:24.884588    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:24.885378 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.885466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-sysctl-d\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.885466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-run-netns\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.885586 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-conf-dir\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.885586 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/931f933b-ce87-44d1-94ac-8dce917c3546-system-cni-dir\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.885586 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gltt\" (UniqueName: \"kubernetes.io/projected/156da910-4b26-4c06-83f9-aaea524edfbb-kube-api-access-4gltt\") pod \"iptables-alerter-g2dhw\" (UID: \"156da910-4b26-4c06-83f9-aaea524edfbb\") " pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:24.885768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-modprobe-d\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.885768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-kubernetes\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.885768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-sys\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.885768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885698    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.885768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-lib-modules\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.885768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.885768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-hostroot\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.885768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/156da910-4b26-4c06-83f9-aaea524edfbb-host-slash\") pod \"iptables-alerter-g2dhw\" (UID: \"156da910-4b26-4c06-83f9-aaea524edfbb\") " pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-sysconfig\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ec6202e-b514-4424-9e32-51f7bcc5cb37-tmp\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-cni-dir\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-registration-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsn49\" (UniqueName: \"kubernetes.io/projected/407d04b3-e8cf-4cf4-8b64-7759152c459c-kube-api-access-rsn49\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885882    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-run\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885925    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7dw\" (UniqueName: \"kubernetes.io/projected/7db784ac-d4f5-4f32-ab91-73e83ea92246-kube-api-access-gd7dw\") pod \"node-resolver-jqm7n\" (UID: \"7db784ac-d4f5-4f32-ab91-73e83ea92246\") " pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-cni-binary-copy\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885986    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.885986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/931f933b-ce87-44d1-94ac-8dce917c3546-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886114    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 19:06:24.886133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886127    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-sys-fs\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-sysctl-conf\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886161    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lgkqq\""
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4hxkk\""
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqt65\" (UniqueName: \"kubernetes.io/projected/7ec6202e-b514-4424-9e32-51f7bcc5cb37-kube-api-access-pqt65\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-system-cni-dir\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/931f933b-ce87-44d1-94ac-8dce917c3546-cni-binary-copy\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886261    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b74tn\""
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/931f933b-ce87-44d1-94ac-8dce917c3546-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886356    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-socket-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-device-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-tuned\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-socket-dir-parent\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-var-lib-cni-bin\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-var-lib-cni-multus\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886511    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnb6\" (UniqueName: \"kubernetes.io/projected/931f933b-ce87-44d1-94ac-8dce917c3546-kube-api-access-bvnb6\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:24.886550    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:24.886881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886567    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-systemd\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-var-lib-kubelet\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-host\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7db784ac-d4f5-4f32-ab91-73e83ea92246-hosts-file\") pod \"node-resolver-jqm7n\" (UID: \"7db784ac-d4f5-4f32-ab91-73e83ea92246\") " pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-var-lib-kubelet\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-run-multus-certs\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886750    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-etc-kubernetes\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxspj\" (UniqueName: \"kubernetes.io/projected/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-kube-api-access-vxspj\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db784ac-d4f5-4f32-ab91-73e83ea92246-tmp-dir\") pod \"node-resolver-jqm7n\" (UID: \"7db784ac-d4f5-4f32-ab91-73e83ea92246\") " pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-cnibin\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-run-k8s-cni-cncf-io\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-daemon-config\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/931f933b-ce87-44d1-94ac-8dce917c3546-cnibin\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/931f933b-ce87-44d1-94ac-8dce917c3546-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.886979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/156da910-4b26-4c06-83f9-aaea524edfbb-iptables-alerter-script\") pod \"iptables-alerter-g2dhw\" (UID: \"156da910-4b26-4c06-83f9-aaea524edfbb\") " pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.887004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.887557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.887375    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.889236 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.889219    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 19:06:24.889695 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.889665    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 19:06:24.889898 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.889880    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 19:06:24.890144 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.890124    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 19:06:24.890424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.890410    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hcjfk\""
Sep 17 19:06:24.890629 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.890599    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 19:06:24.890751 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.890730    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 19:06:24.890895 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.890841    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:24.928555 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.928536    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:24.933004 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.932983    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-452rg"
Sep 17 19:06:24.943929 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.943906    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-452rg"
Sep 17 19:06:24.977132 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.977110    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 19:06:24.987921 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.987901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-registration-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.988024 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.987925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-cni-binary-copy\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988024 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.987942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-sys-fs\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.988024 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.987962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-sysctl-conf\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.988024 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.987987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-system-cni-dir\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988024 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/931f933b-ce87-44d1-94ac-8dce917c3546-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsh6p\" (UniqueName: \"kubernetes.io/projected/5da89ee3-23c3-4a18-a046-0dde553add07-kube-api-access-dsh6p\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-registration-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988052    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-sys-fs\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-run-netns\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988100    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-run-systemd\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-system-cni-dir\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-socket-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnb6\" (UniqueName: \"kubernetes.io/projected/931f933b-ce87-44d1-94ac-8dce917c3546-kube-api-access-bvnb6\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-log-socket\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-cni-bin\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-sysctl-conf\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/931f933b-ce87-44d1-94ac-8dce917c3546-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-systemd\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.988244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988232    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-socket-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-systemd\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-run-multus-certs\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-run-multus-certs\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-etc-kubernetes\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxspj\" (UniqueName: \"kubernetes.io/projected/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-kube-api-access-vxspj\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/156da910-4b26-4c06-83f9-aaea524edfbb-host-slash\") pod \"iptables-alerter-g2dhw\" (UID: \"156da910-4b26-4c06-83f9-aaea524edfbb\") " pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988361    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151-host\") pod \"node-ca-8fmwv\" (UID: \"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151\") " pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-etc-kubernetes\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/156da910-4b26-4c06-83f9-aaea524edfbb-host-slash\") pod \"iptables-alerter-g2dhw\" (UID: \"156da910-4b26-4c06-83f9-aaea524edfbb\") " pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-systemd-units\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-cni-netd\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-run-k8s-cni-cncf-io\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/931f933b-ce87-44d1-94ac-8dce917c3546-cnibin\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988581    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-cni-binary-copy\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/931f933b-ce87-44d1-94ac-8dce917c3546-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/931f933b-ce87-44d1-94ac-8dce917c3546-cnibin\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.988970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-run-k8s-cni-cncf-io\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/156da910-4b26-4c06-83f9-aaea524edfbb-iptables-alerter-script\") pod \"iptables-alerter-g2dhw\" (UID: \"156da910-4b26-4c06-83f9-aaea524edfbb\") " pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-run-netns\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-conf-dir\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-modprobe-d\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-kubernetes\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-conf-dir\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-hostroot\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-var-lib-openvswitch\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988909    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-kubernetes\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-sysconfig\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-hostroot\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988824    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-run-netns\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-sysconfig\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-cni-dir\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.989805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.988966    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-modprobe-d\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsn49\" (UniqueName: \"kubernetes.io/projected/407d04b3-e8cf-4cf4-8b64-7759152c459c-kube-api-access-rsn49\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-run\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-cni-dir\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7dw\" (UniqueName: \"kubernetes.io/projected/7db784ac-d4f5-4f32-ab91-73e83ea92246-kube-api-access-gd7dw\") pod \"node-resolver-jqm7n\" (UID: \"7db784ac-d4f5-4f32-ab91-73e83ea92246\") " pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/931f933b-ce87-44d1-94ac-8dce917c3546-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/156da910-4b26-4c06-83f9-aaea524edfbb-iptables-alerter-script\") pod \"iptables-alerter-g2dhw\" (UID: \"156da910-4b26-4c06-83f9-aaea524edfbb\") " pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/931f933b-ce87-44d1-94ac-8dce917c3546-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989169    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-run\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c0c7c61-2325-402e-a299-0b24efefe9fd-ovn-node-metrics-cert\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c0c7c61-2325-402e-a299-0b24efefe9fd-ovnkube-script-lib\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqt65\" (UniqueName: \"kubernetes.io/projected/7ec6202e-b514-4424-9e32-51f7bcc5cb37-kube-api-access-pqt65\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/931f933b-ce87-44d1-94ac-8dce917c3546-cni-binary-copy\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5e92de3-419a-4bff-858a-d15f9509c77a-kubelet-config\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-device-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-tuned\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-socket-dir-parent\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.990514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-var-lib-cni-bin\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-var-lib-cni-multus\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-device-dir\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-kubelet\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-var-lib-cni-bin\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-var-lib-cni-multus\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-socket-dir-parent\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-slash\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-etc-openvswitch\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-var-lib-kubelet\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-host\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989677    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-var-lib-kubelet\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7db784ac-d4f5-4f32-ab91-73e83ea92246-hosts-file\") pod \"node-resolver-jqm7n\" (UID: \"7db784ac-d4f5-4f32-ab91-73e83ea92246\") " pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-var-lib-kubelet\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/931f933b-ce87-44d1-94ac-8dce917c3546-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-host\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4afed170-1f22-445b-b281-d4534939b551-konnectivity-ca\") pod \"konnectivity-agent-s7wdr\" (UID: \"4afed170-1f22-445b-b281-d4534939b551\") " pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989738    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 19:06:24.991164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-host-var-lib-kubelet\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7db784ac-d4f5-4f32-ab91-73e83ea92246-hosts-file\") pod \"node-resolver-jqm7n\" (UID: \"7db784ac-d4f5-4f32-ab91-73e83ea92246\") " pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/931f933b-ce87-44d1-94ac-8dce917c3546-cni-binary-copy\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-run-openvswitch\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.989894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db784ac-d4f5-4f32-ab91-73e83ea92246-tmp-dir\") pod \"node-resolver-jqm7n\" (UID: \"7db784ac-d4f5-4f32-ab91-73e83ea92246\") " pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db784ac-d4f5-4f32-ab91-73e83ea92246-tmp-dir\") pod \"node-resolver-jqm7n\" (UID: \"7db784ac-d4f5-4f32-ab91-73e83ea92246\") " pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-cnibin\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-daemon-config\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4afed170-1f22-445b-b281-d4534939b551-agent-certs\") pod \"konnectivity-agent-s7wdr\" (UID: \"4afed170-1f22-445b-b281-d4534939b551\") " pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-multus-daemon-config\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151-serviceca\") pod \"node-ca-8fmwv\" (UID: \"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151\") " pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990680    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-run-ovn\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990741    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-sysctl-d\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/931f933b-ce87-44d1-94ac-8dce917c3546-system-cni-dir\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.992019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gltt\" (UniqueName: \"kubernetes.io/projected/156da910-4b26-4c06-83f9-aaea524edfbb-kube-api-access-4gltt\") pod \"iptables-alerter-g2dhw\" (UID: \"156da910-4b26-4c06-83f9-aaea524edfbb\") " pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.990812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-node-log\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwwt\" (UniqueName: \"kubernetes.io/projected/9c0c7c61-2325-402e-a299-0b24efefe9fd-kube-api-access-rfwwt\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991244    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-sys\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-cnibin\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-lib-modules\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5e92de3-419a-4bff-858a-d15f9509c77a-dbus\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/407d04b3-e8cf-4cf4-8b64-7759152c459c-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-lib-modules\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991441    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-sys\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-sysctl-d\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991476    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/931f933b-ce87-44d1-94ac-8dce917c3546-system-cni-dir\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991538    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw5jj\" (UniqueName: \"kubernetes.io/projected/14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151-kube-api-access-nw5jj\") pod \"node-ca-8fmwv\" (UID: \"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151\") " pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c0c7c61-2325-402e-a299-0b24efefe9fd-ovnkube-config\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c0c7c61-2325-402e-a299-0b24efefe9fd-env-overrides\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:24.992720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.991638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ec6202e-b514-4424-9e32-51f7bcc5cb37-tmp\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.994154 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.994122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ec6202e-b514-4424-9e32-51f7bcc5cb37-tmp\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:24.996802 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:24.996777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ec6202e-b514-4424-9e32-51f7bcc5cb37-etc-tuned\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:25.007287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.007000    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqt65\" (UniqueName: \"kubernetes.io/projected/7ec6202e-b514-4424-9e32-51f7bcc5cb37-kube-api-access-pqt65\") pod \"tuned-p6pfc\" (UID: \"7ec6202e-b514-4424-9e32-51f7bcc5cb37\") " pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:25.009445 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.009377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7dw\" (UniqueName: \"kubernetes.io/projected/7db784ac-d4f5-4f32-ab91-73e83ea92246-kube-api-access-gd7dw\") pod \"node-resolver-jqm7n\" (UID: \"7db784ac-d4f5-4f32-ab91-73e83ea92246\") " pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:25.010115 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.010092    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gltt\" (UniqueName: \"kubernetes.io/projected/156da910-4b26-4c06-83f9-aaea524edfbb-kube-api-access-4gltt\") pod \"iptables-alerter-g2dhw\" (UID: \"156da910-4b26-4c06-83f9-aaea524edfbb\") " pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:25.010209 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.010091    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsn49\" (UniqueName: \"kubernetes.io/projected/407d04b3-e8cf-4cf4-8b64-7759152c459c-kube-api-access-rsn49\") pod \"aws-ebs-csi-driver-node-r6hhm\" (UID: \"407d04b3-e8cf-4cf4-8b64-7759152c459c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:25.011086 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.011068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxspj\" (UniqueName: \"kubernetes.io/projected/9147622f-fd81-4ea0-a693-7e4d0dab9d0b-kube-api-access-vxspj\") pod \"multus-7mb94\" (UID: \"9147622f-fd81-4ea0-a693-7e4d0dab9d0b\") " pod="openshift-multus/multus-7mb94"
Sep 17 19:06:25.011420 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.011398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnb6\" (UniqueName: \"kubernetes.io/projected/931f933b-ce87-44d1-94ac-8dce917c3546-kube-api-access-bvnb6\") pod \"multus-additional-cni-plugins-vp9lm\" (UID: \"931f933b-ce87-44d1-94ac-8dce917c3546\") " pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:25.037301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.037257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"4eb197abc8b394d582a58b08fa210ad1","Type":"ContainerStarted","Data":"3ea24a090456deb909db972347ae83524c091a364895ed75b2d3504176788cb6"}
Sep 17 19:06:25.038577 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.038552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"8ef18637fd00ca6d2dbdae6b5266649f","Type":"ContainerStarted","Data":"5c027271a78dbe7442bba6e729dc8b2b483a696a946e7b99041c395bf21ba169"}
Sep 17 19:06:25.092295 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-var-lib-openvswitch\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c0c7c61-2325-402e-a299-0b24efefe9fd-ovn-node-metrics-cert\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c0c7c61-2325-402e-a299-0b24efefe9fd-ovnkube-script-lib\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5e92de3-419a-4bff-858a-d15f9509c77a-kubelet-config\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-kubelet\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-slash\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092373    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-var-lib-openvswitch\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-etc-openvswitch\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4afed170-1f22-445b-b281-d4534939b551-konnectivity-ca\") pod \"konnectivity-agent-s7wdr\" (UID: \"4afed170-1f22-445b-b281-d4534939b551\") " pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-kubelet\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-run-openvswitch\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092506    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-etc-openvswitch\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5e92de3-419a-4bff-858a-d15f9509c77a-kubelet-config\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092522    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-run-openvswitch\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-slash\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4afed170-1f22-445b-b281-d4534939b551-agent-certs\") pod \"konnectivity-agent-s7wdr\" (UID: \"4afed170-1f22-445b-b281-d4534939b551\") " pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151-serviceca\") pod \"node-ca-8fmwv\" (UID: \"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151\") " pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-run-ovn\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-node-log\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092735    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.092747    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwwt\" (UniqueName: \"kubernetes.io/projected/9c0c7c61-2325-402e-a299-0b24efefe9fd-kube-api-access-rfwwt\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5e92de3-419a-4bff-858a-d15f9509c77a-dbus\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.092837    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs podName:5da89ee3-23c3-4a18-a046-0dde553add07 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:25.592797118 +0000 UTC m=+4.720538501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs") pod "network-metrics-daemon-g886c" (UID: "5da89ee3-23c3-4a18-a046-0dde553add07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:25.092983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5e92de3-419a-4bff-858a-d15f9509c77a-dbus\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092891    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-run-ovn\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-node-log\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.092940    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.092976    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret podName:b5e92de3-419a-4bff-858a-d15f9509c77a nodeName:}" failed. No retries permitted until 2026-09-17 19:06:25.59296393 +0000 UTC m=+4.720705315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret") pod "global-pull-secret-syncer-xhl95" (UID: "b5e92de3-419a-4bff-858a-d15f9509c77a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.092973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4afed170-1f22-445b-b281-d4534939b551-konnectivity-ca\") pod \"konnectivity-agent-s7wdr\" (UID: \"4afed170-1f22-445b-b281-d4534939b551\") " pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151-serviceca\") pod \"node-ca-8fmwv\" (UID: \"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151\") " pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5jj\" (UniqueName: \"kubernetes.io/projected/14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151-kube-api-access-nw5jj\") pod \"node-ca-8fmwv\" (UID: \"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151\") " pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c0c7c61-2325-402e-a299-0b24efefe9fd-ovnkube-config\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c0c7c61-2325-402e-a299-0b24efefe9fd-env-overrides\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c0c7c61-2325-402e-a299-0b24efefe9fd-ovnkube-script-lib\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsh6p\" (UniqueName: \"kubernetes.io/projected/5da89ee3-23c3-4a18-a046-0dde553add07-kube-api-access-dsh6p\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-run-netns\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-run-systemd\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-log-socket\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.093776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-cni-bin\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151-host\") pod \"node-ca-8fmwv\" (UID: \"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151\") " pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-run-systemd\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-systemd-units\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-log-socket\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-run-netns\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-cni-netd\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-systemd-units\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151-host\") pod \"node-ca-8fmwv\" (UID: \"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151\") " pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093642    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c0c7c61-2325-402e-a299-0b24efefe9fd-ovnkube-config\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-cni-bin\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093672    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c0c7c61-2325-402e-a299-0b24efefe9fd-env-overrides\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.094485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.093685    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c0c7c61-2325-402e-a299-0b24efefe9fd-host-cni-netd\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.095035 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.095024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c0c7c61-2325-402e-a299-0b24efefe9fd-ovn-node-metrics-cert\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.095258 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.095242    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4afed170-1f22-445b-b281-d4534939b551-agent-certs\") pod \"konnectivity-agent-s7wdr\" (UID: \"4afed170-1f22-445b-b281-d4534939b551\") " pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:25.107080 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.107054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw5jj\" (UniqueName: \"kubernetes.io/projected/14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151-kube-api-access-nw5jj\") pod \"node-ca-8fmwv\" (UID: \"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151\") " pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:25.109852 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.109793    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:25.109852 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.109818    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:25.109852 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.109833    2583 projected.go:196] Error preparing data for projected volume kube-api-access-vvd62 for pod openshift-network-diagnostics/network-check-target-vdcz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:25.110073 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.109904    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62 podName:abc81dc4-5b90-4a46-ad49-35d0fb866226 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:25.609887769 +0000 UTC m=+4.737629163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vvd62" (UniqueName: "kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62") pod "network-check-target-vdcz8" (UID: "abc81dc4-5b90-4a46-ad49-35d0fb866226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:25.112275 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.112248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwwt\" (UniqueName: \"kubernetes.io/projected/9c0c7c61-2325-402e-a299-0b24efefe9fd-kube-api-access-rfwwt\") pod \"ovnkube-node-jtf4n\" (UID: \"9c0c7c61-2325-402e-a299-0b24efefe9fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.112402 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.112381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsh6p\" (UniqueName: \"kubernetes.io/projected/5da89ee3-23c3-4a18-a046-0dde553add07-kube-api-access-dsh6p\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:25.179686 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.179653    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm"
Sep 17 19:06:25.187874 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.187852    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p6pfc"
Sep 17 19:06:25.195190 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:25.195169    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec6202e_b514_4424_9e32_51f7bcc5cb37.slice/crio-51a7403691def630af7f47754c072150c82ef4617d1f88de6df9a806e239506a WatchSource:0}: Error finding container 51a7403691def630af7f47754c072150c82ef4617d1f88de6df9a806e239506a: Status 404 returned error can't find the container with id 51a7403691def630af7f47754c072150c82ef4617d1f88de6df9a806e239506a
Sep 17 19:06:25.196110 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.196090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jqm7n"
Sep 17 19:06:25.201558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.201531    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:25.203329 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:25.203245    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db784ac_d4f5_4f32_ab91_73e83ea92246.slice/crio-ae84d0ce12012bf2196723457e0991940474d94b21c375c1c359b0a91d549287 WatchSource:0}: Error finding container ae84d0ce12012bf2196723457e0991940474d94b21c375c1c359b0a91d549287: Status 404 returned error can't find the container with id ae84d0ce12012bf2196723457e0991940474d94b21c375c1c359b0a91d549287
Sep 17 19:06:25.208832 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.208810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7mb94"
Sep 17 19:06:25.210667 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:25.209598    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0c7c61_2325_402e_a299_0b24efefe9fd.slice/crio-52cb8a10d60b01ff5e4e8474c75292dd9ac0e0b1543de5f41cc9fd256df80c2c WatchSource:0}: Error finding container 52cb8a10d60b01ff5e4e8474c75292dd9ac0e0b1543de5f41cc9fd256df80c2c: Status 404 returned error can't find the container with id 52cb8a10d60b01ff5e4e8474c75292dd9ac0e0b1543de5f41cc9fd256df80c2c
Sep 17 19:06:25.215249 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.215228    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vp9lm"
Sep 17 19:06:25.219368 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:25.219093    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9147622f_fd81_4ea0_a693_7e4d0dab9d0b.slice/crio-74d4a835652ad74671d97fed922f4dc7bd6e2f6ce7627beeb8f8d7859a25d1e9 WatchSource:0}: Error finding container 74d4a835652ad74671d97fed922f4dc7bd6e2f6ce7627beeb8f8d7859a25d1e9: Status 404 returned error can't find the container with id 74d4a835652ad74671d97fed922f4dc7bd6e2f6ce7627beeb8f8d7859a25d1e9
Sep 17 19:06:25.222080 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.222058    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g2dhw"
Sep 17 19:06:25.224480 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:25.224463    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931f933b_ce87_44d1_94ac_8dce917c3546.slice/crio-79c5d4b52f3168f139fcc572a2a55f0e25caf1ba87d8ac157441cc3b065360cf WatchSource:0}: Error finding container 79c5d4b52f3168f139fcc572a2a55f0e25caf1ba87d8ac157441cc3b065360cf: Status 404 returned error can't find the container with id 79c5d4b52f3168f139fcc572a2a55f0e25caf1ba87d8ac157441cc3b065360cf
Sep 17 19:06:25.227473 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.227448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:25.230291 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:25.230270    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156da910_4b26_4c06_83f9_aaea524edfbb.slice/crio-c888a42e7ee9aecf40e02ac83f35708dc8244975f275d228fc1f4a7af590bae5 WatchSource:0}: Error finding container c888a42e7ee9aecf40e02ac83f35708dc8244975f275d228fc1f4a7af590bae5: Status 404 returned error can't find the container with id c888a42e7ee9aecf40e02ac83f35708dc8244975f275d228fc1f4a7af590bae5
Sep 17 19:06:25.232296 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.232005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8fmwv"
Sep 17 19:06:25.242598 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:25.238656    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afed170_1f22_445b_b281_d4534939b551.slice/crio-8ae0050a1092dccfdb3bb6793180687b3325c9ab7b33dcf5416bb02e5b2e0acb WatchSource:0}: Error finding container 8ae0050a1092dccfdb3bb6793180687b3325c9ab7b33dcf5416bb02e5b2e0acb: Status 404 returned error can't find the container with id 8ae0050a1092dccfdb3bb6793180687b3325c9ab7b33dcf5416bb02e5b2e0acb
Sep 17 19:06:25.599445 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.598697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:25.599445 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.598749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:25.599445 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.598919    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:25.599445 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.598982    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret podName:b5e92de3-419a-4bff-858a-d15f9509c77a nodeName:}" failed. No retries permitted until 2026-09-17 19:06:26.598961668 +0000 UTC m=+5.726703052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret") pod "global-pull-secret-syncer-xhl95" (UID: "b5e92de3-419a-4bff-858a-d15f9509c77a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:25.599445 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.599359    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:25.599445 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.599409    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs podName:5da89ee3-23c3-4a18-a046-0dde553add07 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:26.599393504 +0000 UTC m=+5.727134889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs") pod "network-metrics-daemon-g886c" (UID: "5da89ee3-23c3-4a18-a046-0dde553add07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:25.700205 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.700156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:25.700396 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.700320    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:25.700396 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.700345    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:25.700396 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.700358    2583 projected.go:196] Error preparing data for projected volume kube-api-access-vvd62 for pod openshift-network-diagnostics/network-check-target-vdcz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:25.700561 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:25.700415    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62 podName:abc81dc4-5b90-4a46-ad49-35d0fb866226 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:26.700397284 +0000 UTC m=+5.828138677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vvd62" (UniqueName: "kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62") pod "network-check-target-vdcz8" (UID: "abc81dc4-5b90-4a46-ad49-35d0fb866226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:25.945900 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.945791    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:24 +0000 UTC" deadline="2028-04-28 16:30:36.36073276 +0000 UTC"
Sep 17 19:06:25.945900 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:25.945850    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14133h24m10.414887313s"
Sep 17 19:06:26.033691 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.033127    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:26.033691 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:26.033259    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:26.057301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.057208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7wdr" event={"ID":"4afed170-1f22-445b-b281-d4534939b551","Type":"ContainerStarted","Data":"8ae0050a1092dccfdb3bb6793180687b3325c9ab7b33dcf5416bb02e5b2e0acb"}
Sep 17 19:06:26.065158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.065096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vp9lm" event={"ID":"931f933b-ce87-44d1-94ac-8dce917c3546","Type":"ContainerStarted","Data":"79c5d4b52f3168f139fcc572a2a55f0e25caf1ba87d8ac157441cc3b065360cf"}
Sep 17 19:06:26.068923 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.068857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7mb94" event={"ID":"9147622f-fd81-4ea0-a693-7e4d0dab9d0b","Type":"ContainerStarted","Data":"74d4a835652ad74671d97fed922f4dc7bd6e2f6ce7627beeb8f8d7859a25d1e9"}
Sep 17 19:06:26.082525 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.082481    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p6pfc" event={"ID":"7ec6202e-b514-4424-9e32-51f7bcc5cb37","Type":"ContainerStarted","Data":"51a7403691def630af7f47754c072150c82ef4617d1f88de6df9a806e239506a"}
Sep 17 19:06:26.089589 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.089556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm" event={"ID":"407d04b3-e8cf-4cf4-8b64-7759152c459c","Type":"ContainerStarted","Data":"368c4d7c1f00aa39d457b3ca0d896cb47e042be7c20681e2431c7f80ba1ff713"}
Sep 17 19:06:26.095038 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.094125    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ef18637fd00ca6d2dbdae6b5266649f" containerID="8ab5d901d6bbd48dd7d9e3d4898a0d860362270707543513e8355a894d525363" exitCode=0
Sep 17 19:06:26.095038 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.094835    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"8ef18637fd00ca6d2dbdae6b5266649f","Type":"ContainerDied","Data":"8ab5d901d6bbd48dd7d9e3d4898a0d860362270707543513e8355a894d525363"}
Sep 17 19:06:26.102286 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.102247    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8fmwv" event={"ID":"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151","Type":"ContainerStarted","Data":"c9af15af11c30b2cfbdc7888fca524694a7cfbcfcad23a29917ee23e11d9d730"}
Sep 17 19:06:26.124408 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.124353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g2dhw" event={"ID":"156da910-4b26-4c06-83f9-aaea524edfbb","Type":"ContainerStarted","Data":"c888a42e7ee9aecf40e02ac83f35708dc8244975f275d228fc1f4a7af590bae5"}
Sep 17 19:06:26.153698 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.152979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerStarted","Data":"52cb8a10d60b01ff5e4e8474c75292dd9ac0e0b1543de5f41cc9fd256df80c2c"}
Sep 17 19:06:26.158663 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.158564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jqm7n" event={"ID":"7db784ac-d4f5-4f32-ab91-73e83ea92246","Type":"ContainerStarted","Data":"ae84d0ce12012bf2196723457e0991940474d94b21c375c1c359b0a91d549287"}
Sep 17 19:06:26.608539 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.607805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:26.608539 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.607854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:26.608539 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:26.607970    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:26.608539 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:26.608028    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret podName:b5e92de3-419a-4bff-858a-d15f9509c77a nodeName:}" failed. No retries permitted until 2026-09-17 19:06:28.608011075 +0000 UTC m=+7.735752455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret") pod "global-pull-secret-syncer-xhl95" (UID: "b5e92de3-419a-4bff-858a-d15f9509c77a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:26.608539 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:26.608427    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:26.608539 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:26.608488    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs podName:5da89ee3-23c3-4a18-a046-0dde553add07 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:28.608470481 +0000 UTC m=+7.736211865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs") pod "network-metrics-daemon-g886c" (UID: "5da89ee3-23c3-4a18-a046-0dde553add07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:26.709191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.709150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:26.709358 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:26.709314    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:26.709358 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:26.709333    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:26.709358 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:26.709346    2583 projected.go:196] Error preparing data for projected volume kube-api-access-vvd62 for pod openshift-network-diagnostics/network-check-target-vdcz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:26.709523 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:26.709404    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62 podName:abc81dc4-5b90-4a46-ad49-35d0fb866226 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:28.709386409 +0000 UTC m=+7.837127793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vvd62" (UniqueName: "kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62") pod "network-check-target-vdcz8" (UID: "abc81dc4-5b90-4a46-ad49-35d0fb866226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:26.946803 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.946500    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:24 +0000 UTC" deadline="2028-03-25 11:13:16.851137928 +0000 UTC"
Sep 17 19:06:26.947234 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:26.946849    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13312h6m49.904297621s"
Sep 17 19:06:27.035454 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:27.034808    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:27.035454 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:27.034937    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:27.035454 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:27.035314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:27.035454 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:27.035414    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:28.033564 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:28.033532    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:28.034003 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:28.033686    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:28.627254 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:28.626543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:28.627254 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:28.626596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:28.627254 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:28.626749    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:28.627254 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:28.626806    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret podName:b5e92de3-419a-4bff-858a-d15f9509c77a nodeName:}" failed. No retries permitted until 2026-09-17 19:06:32.626788946 +0000 UTC m=+11.754530330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret") pod "global-pull-secret-syncer-xhl95" (UID: "b5e92de3-419a-4bff-858a-d15f9509c77a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:28.627254 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:28.627158    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:28.627254 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:28.627201    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs podName:5da89ee3-23c3-4a18-a046-0dde553add07 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:32.627185994 +0000 UTC m=+11.754927377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs") pod "network-metrics-daemon-g886c" (UID: "5da89ee3-23c3-4a18-a046-0dde553add07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:28.727930 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:28.727893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:28.728080 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:28.728054    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:28.728080 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:28.728074    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:28.728170 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:28.728090    2583 projected.go:196] Error preparing data for projected volume kube-api-access-vvd62 for pod openshift-network-diagnostics/network-check-target-vdcz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:28.728170 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:28.728142    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62 podName:abc81dc4-5b90-4a46-ad49-35d0fb866226 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:32.728125602 +0000 UTC m=+11.855866985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vvd62" (UniqueName: "kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62") pod "network-check-target-vdcz8" (UID: "abc81dc4-5b90-4a46-ad49-35d0fb866226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:29.035329 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:29.034723    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:29.035329 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:29.034860    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:29.035329 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:29.034723    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:29.035329 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:29.035264    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:30.033328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:30.033299    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:30.033494 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:30.033406    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:31.034378 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:31.033729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:31.034378 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:31.033883    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:31.034378 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:31.034222    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:31.034378 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:31.034338    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:32.033549 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:32.033510    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:32.033753 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:32.033669    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:32.661628 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:32.661577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:32.662061 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:32.661650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:32.662061 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:32.661778    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:32.662061 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:32.661844    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret podName:b5e92de3-419a-4bff-858a-d15f9509c77a nodeName:}" failed. No retries permitted until 2026-09-17 19:06:40.66182254 +0000 UTC m=+19.789563938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret") pod "global-pull-secret-syncer-xhl95" (UID: "b5e92de3-419a-4bff-858a-d15f9509c77a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:32.662333 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:32.662242    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:32.662333 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:32.662300    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs podName:5da89ee3-23c3-4a18-a046-0dde553add07 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:40.662283275 +0000 UTC m=+19.790024661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs") pod "network-metrics-daemon-g886c" (UID: "5da89ee3-23c3-4a18-a046-0dde553add07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:32.762806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:32.762768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:32.763037 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:32.763015    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:32.763095 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:32.763048    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:32.763095 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:32.763061    2583 projected.go:196] Error preparing data for projected volume kube-api-access-vvd62 for pod openshift-network-diagnostics/network-check-target-vdcz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:32.763196 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:32.763128    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62 podName:abc81dc4-5b90-4a46-ad49-35d0fb866226 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:40.76310905 +0000 UTC m=+19.890850443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vvd62" (UniqueName: "kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62") pod "network-check-target-vdcz8" (UID: "abc81dc4-5b90-4a46-ad49-35d0fb866226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:33.034221 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:33.034191    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:33.034440 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:33.034273    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:33.034440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:33.034365    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:33.034513 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:33.034490    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:34.033506 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:34.033469    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:34.033970 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:34.033600    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:35.033959 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:35.033928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:35.034387 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:35.034082    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:35.034387 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:35.034106    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:35.034387 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:35.034227    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:36.033185 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:36.033110    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:36.033346 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:36.033234    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:37.033946 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:37.033910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:37.034324 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:37.034054    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:37.034324 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:37.034116    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:37.034324 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:37.034236    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:38.033344 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:38.033313    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:38.033509 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:38.033429    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:39.033658 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:39.033608    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:39.034090 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:39.033761    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:39.034090 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:39.033816    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:39.034090 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:39.033934    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:40.033781 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:40.033744    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:40.034245 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:40.033874    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:40.724049 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:40.724015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:40.724234 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:40.724096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:40.724234 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:40.724186    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:40.724234 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:40.724185    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:40.724369 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:40.724245    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs podName:5da89ee3-23c3-4a18-a046-0dde553add07 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.72422974 +0000 UTC m=+35.851971123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs") pod "network-metrics-daemon-g886c" (UID: "5da89ee3-23c3-4a18-a046-0dde553add07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:40.724369 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:40.724258    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret podName:b5e92de3-419a-4bff-858a-d15f9509c77a nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.724252087 +0000 UTC m=+35.851993468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret") pod "global-pull-secret-syncer-xhl95" (UID: "b5e92de3-419a-4bff-858a-d15f9509c77a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:40.824426 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:40.824394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:40.824605 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:40.824559    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:40.824605 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:40.824581    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:40.824605 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:40.824595    2583 projected.go:196] Error preparing data for projected volume kube-api-access-vvd62 for pod openshift-network-diagnostics/network-check-target-vdcz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:40.824785 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:40.824677    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62 podName:abc81dc4-5b90-4a46-ad49-35d0fb866226 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.824657156 +0000 UTC m=+35.952398553 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vvd62" (UniqueName: "kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62") pod "network-check-target-vdcz8" (UID: "abc81dc4-5b90-4a46-ad49-35d0fb866226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:41.033034 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:41.032952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:41.033034 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:41.032975    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:41.033250 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:41.033079    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:41.033250 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:41.033198    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:42.033007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:42.032975    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:42.033451 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:42.033085    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:43.034334 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:43.034309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:43.034698 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:43.034404    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:43.034698 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:43.034481    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:43.034698 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:43.034575    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:44.033338 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.033141    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:44.033527 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:44.033432    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:44.199035 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.199001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm" event={"ID":"407d04b3-e8cf-4cf4-8b64-7759152c459c","Type":"ContainerStarted","Data":"2210223294c8940a655689c265eefef854e202224b1565f4de6d2c4d3f1008a0"}
Sep 17 19:06:44.200886 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.200863    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"8ef18637fd00ca6d2dbdae6b5266649f","Type":"ContainerStarted","Data":"a8900bff01015caad13a6d1576f6ef09614f4bba3c5ac67de31324561fc7ab6a"}
Sep 17 19:06:44.202319 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.202294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8fmwv" event={"ID":"14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151","Type":"ContainerStarted","Data":"34e1f6bffbc6fdf9322ed031375906bb15d35b216abe7328aeba4214a9cffc51"}
Sep 17 19:06:44.204791 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.204773    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:06:44.205111 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.205089    2583 generic.go:356] "Generic (PLEG): container finished" podID="9c0c7c61-2325-402e-a299-0b24efefe9fd" containerID="a4b3231364cc1714eeefe1a569872be9ea27d50eeb44bc1ed9f5c63001a2373a" exitCode=1
Sep 17 19:06:44.205190 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.205155    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerStarted","Data":"eca10c2c4ccfef62c3e02eb9f2f849f99b28d251aff7fb62e4c5295b69558ad1"}
Sep 17 19:06:44.205240 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.205219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerStarted","Data":"c3c4c2aea8e7fa9662e98ba0017f10992b60ec8d5978a62e667482cde768913f"}
Sep 17 19:06:44.205278 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.205259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerStarted","Data":"bc79a16d88d4d732cf190de5e976fb02bc9e3d940cf1f49777a9819e1513c648"}
Sep 17 19:06:44.205314 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.205288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerStarted","Data":"58d01bb3e35dada16f745079121025554e6a03b78f931507f28abfdef4b92a56"}
Sep 17 19:06:44.205344 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.205320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerDied","Data":"a4b3231364cc1714eeefe1a569872be9ea27d50eeb44bc1ed9f5c63001a2373a"}
Sep 17 19:06:44.205379 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.205353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerStarted","Data":"cf78d982be6b35ef10c054ab0ef68b42f86c90246f88cd81c0a10376b8f86fc9"}
Sep 17 19:06:44.206464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.206447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jqm7n" event={"ID":"7db784ac-d4f5-4f32-ab91-73e83ea92246","Type":"ContainerStarted","Data":"8daa121e54ed7d3e677154fd87cd07fe691033cd72253f01c68bd552c23ca553"}
Sep 17 19:06:44.207934 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.207909    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"4eb197abc8b394d582a58b08fa210ad1","Type":"ContainerStarted","Data":"7700eb3ea08e1fbb28200cd7061d42929cd7062c6786e420fed3ea11c70e578f"}
Sep 17 19:06:44.209160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.209144    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7wdr" event={"ID":"4afed170-1f22-445b-b281-d4534939b551","Type":"ContainerStarted","Data":"7bf0943f10d15dcedeb65a5b818601bc5366a660b0716be89fd9ec1545053d98"}
Sep 17 19:06:44.210509 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.210487    2583 generic.go:356] "Generic (PLEG): container finished" podID="931f933b-ce87-44d1-94ac-8dce917c3546" containerID="d0d1178f154f472b64a86b17661ba8419f65e7d330b4c8e031217853ef6a8ffd" exitCode=0
Sep 17 19:06:44.210606 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.210562    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vp9lm" event={"ID":"931f933b-ce87-44d1-94ac-8dce917c3546","Type":"ContainerDied","Data":"d0d1178f154f472b64a86b17661ba8419f65e7d330b4c8e031217853ef6a8ffd"}
Sep 17 19:06:44.212049 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.212019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7mb94" event={"ID":"9147622f-fd81-4ea0-a693-7e4d0dab9d0b","Type":"ContainerStarted","Data":"35a563b37d5a12718726d6f4ea7d028567032f7ce8d200ab73a559529637c3ea"}
Sep 17 19:06:44.213652 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.213633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p6pfc" event={"ID":"7ec6202e-b514-4424-9e32-51f7bcc5cb37","Type":"ContainerStarted","Data":"24658f444abf7284eaafa2763afd2c4002837916e5f0c290b0cbdc0596045392"}
Sep 17 19:06:44.218346 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.218312    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" podStartSLOduration=21.218302551 podStartE2EDuration="21.218302551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.218252251 +0000 UTC m=+23.345993652" watchObservedRunningTime="2026-09-17 19:06:44.218302551 +0000 UTC m=+23.346043951"
Sep 17 19:06:44.237541 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.237505    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7mb94" podStartSLOduration=3.322032774 podStartE2EDuration="21.237493678s" totalImagesPullingTime="17.915460904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.237100124 +0000 UTC m=+23.364841523" watchObservedRunningTime="2026-09-17 19:06:44.237493678 +0000 UTC m=+23.365235078"
Sep 17 19:06:44.254029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.253991    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8fmwv" podStartSLOduration=3.413834645 podStartE2EDuration="21.253980104s" totalImagesPullingTime="17.840145459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.253465128 +0000 UTC m=+23.381206526" watchObservedRunningTime="2026-09-17 19:06:44.253980104 +0000 UTC m=+23.381721505"
Sep 17 19:06:44.271833 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.271796    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" podStartSLOduration=21.271786775 podStartE2EDuration="21.271786775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.271468334 +0000 UTC m=+23.399209734" watchObservedRunningTime="2026-09-17 19:06:44.271786775 +0000 UTC m=+23.399528175"
Sep 17 19:06:44.310237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.310197    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jqm7n" podStartSLOduration=3.431045077 podStartE2EDuration="21.310185402s" totalImagesPullingTime="17.879140325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.289583244 +0000 UTC m=+23.417324646" watchObservedRunningTime="2026-09-17 19:06:44.310185402 +0000 UTC m=+23.437926800"
Sep 17 19:06:44.325142 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.325108    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s7wdr" podStartSLOduration=3.481512693 podStartE2EDuration="21.325096463s" totalImagesPullingTime="17.84358377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.325044066 +0000 UTC m=+23.452785467" watchObservedRunningTime="2026-09-17 19:06:44.325096463 +0000 UTC m=+23.452837863"
Sep 17 19:06:44.342029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.341984    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p6pfc" podStartSLOduration=3.451871178 podStartE2EDuration="21.341969146s" totalImagesPullingTime="17.890097968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.341969978 +0000 UTC m=+23.469711378" watchObservedRunningTime="2026-09-17 19:06:44.341969146 +0000 UTC m=+23.469710548"
Sep 17 19:06:44.621999 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.621949    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 19:06:44.968947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.968820    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T19:06:44.621966962Z","UUID":"acd5c468-517d-4ed0-bab6-bec04d5a7427","Handler":null,"Name":"","Endpoint":""}
Sep 17 19:06:44.970581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.970559    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 19:06:44.970581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:44.970596    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 19:06:45.033773 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:45.033742    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:45.033947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:45.033798    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:45.034016 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:45.033942    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:45.034200 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:45.034178    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:45.219201 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:45.219127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm" event={"ID":"407d04b3-e8cf-4cf4-8b64-7759152c459c","Type":"ContainerStarted","Data":"cdf0375eb6ed2668013f23d0a68c0d0cf1a6fabf84ad8a5ca14db939fdad36d2"}
Sep 17 19:06:45.221102 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:45.221064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g2dhw" event={"ID":"156da910-4b26-4c06-83f9-aaea524edfbb","Type":"ContainerStarted","Data":"04891accfb7a410e196b95a0fde95de1ae5b630cef816ad49c508aa7b268ee4d"}
Sep 17 19:06:46.033390 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:46.033355    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:46.036771 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:46.033804    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:46.225571 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:46.225542    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:06:46.225989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:46.225961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerStarted","Data":"e7c18d276eb5c12b1372b480538750b4afaffdeec9adf7f8afab1067d6f473c5"}
Sep 17 19:06:46.227917 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:46.227890    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm" event={"ID":"407d04b3-e8cf-4cf4-8b64-7759152c459c","Type":"ContainerStarted","Data":"69e0b08a8f401843190c4d0ca65a1e4b7f9bfbeecbd3523b743f8cb15be0049a"}
Sep 17 19:06:46.248021 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:46.247976    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g2dhw" podStartSLOduration=5.394178532 podStartE2EDuration="23.247964157s" totalImagesPullingTime="17.853785625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:45.238167602 +0000 UTC m=+24.365909001" watchObservedRunningTime="2026-09-17 19:06:46.247964157 +0000 UTC m=+25.375705559"
Sep 17 19:06:47.033056 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:47.033021    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:47.033056 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:47.033058    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:47.033319 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:47.033162    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:47.033319 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:47.033293    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:47.749800 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:47.749768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:47.750413 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:47.750388    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:47.769337 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:47.769299    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6hhm" podStartSLOduration=8.6482371 podStartE2EDuration="24.769284074s" totalImagesPullingTime="16.121046974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:46.24906976 +0000 UTC m=+25.376811161" watchObservedRunningTime="2026-09-17 19:06:47.769284074 +0000 UTC m=+26.897025875"
Sep 17 19:06:48.033515 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:48.033441    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:48.033685 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:48.033562    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:48.743526 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:48.743347    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:48.743975 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:48.743955    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s7wdr"
Sep 17 19:06:49.033568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:49.033538    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:49.033568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:49.033562    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:49.034312 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:49.033665    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:49.034312 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:49.033806    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:49.237069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:49.237038    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:06:49.237448 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:49.237428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerStarted","Data":"a74980a817d53866bd5bbc8f9d798740ac01d96ae3b6055424420e81110e4d44"}
Sep 17 19:06:49.237680 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:49.237656    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:49.237894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:49.237879    2583 scope.go:117] "RemoveContainer" containerID="a4b3231364cc1714eeefe1a569872be9ea27d50eeb44bc1ed9f5c63001a2373a"
Sep 17 19:06:49.239527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:49.239502    2583 generic.go:356] "Generic (PLEG): container finished" podID="931f933b-ce87-44d1-94ac-8dce917c3546" containerID="24da60982b15a6cce26e14d82524bfb6cef6c23ed5fac7ad851dc30479d418e2" exitCode=0
Sep 17 19:06:49.239633 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:49.239573    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vp9lm" event={"ID":"931f933b-ce87-44d1-94ac-8dce917c3546","Type":"ContainerDied","Data":"24da60982b15a6cce26e14d82524bfb6cef6c23ed5fac7ad851dc30479d418e2"}
Sep 17 19:06:49.255456 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:49.255435    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:50.033935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.033910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:50.034306 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:50.034008    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:50.246172 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.245880    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:06:50.246477 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.246452    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" event={"ID":"9c0c7c61-2325-402e-a299-0b24efefe9fd","Type":"ContainerStarted","Data":"d36bebe2970b75b9b47e33781ab24aeda95d0a831b56a02c4292f9fb145d8b66"}
Sep 17 19:06:50.246675 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.246647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:50.246829 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.246812    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:50.265132 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.265108    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:06:50.284077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.283805    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n" podStartSLOduration=9.411271158 podStartE2EDuration="27.283786816s" totalImagesPullingTime="17.872515658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:50.282482019 +0000 UTC m=+29.410223423" watchObservedRunningTime="2026-09-17 19:06:50.283786816 +0000 UTC m=+29.411528218"
Sep 17 19:06:50.701267 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.701235    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdcz8"]
Sep 17 19:06:50.701421 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.701357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:50.701457 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:50.701430    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:50.708294 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.708265    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhl95"]
Sep 17 19:06:50.708432 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.708417    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:50.708650 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:50.708525    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:50.712172 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.712147    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g886c"]
Sep 17 19:06:50.712284 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:50.712271    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:50.712415 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:50.712391    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:51.251028 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:51.250988    2583 generic.go:356] "Generic (PLEG): container finished" podID="931f933b-ce87-44d1-94ac-8dce917c3546" containerID="39f85aea6b3d8185be791fe91085edd65c18890be8cddb17ad9cf4da7d8d4670" exitCode=0
Sep 17 19:06:51.251427 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:51.251081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vp9lm" event={"ID":"931f933b-ce87-44d1-94ac-8dce917c3546","Type":"ContainerDied","Data":"39f85aea6b3d8185be791fe91085edd65c18890be8cddb17ad9cf4da7d8d4670"}
Sep 17 19:06:52.033632 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:52.033593    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:52.033811 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:52.033722    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:53.037167 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:53.034973    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:53.037167 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:53.035130    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:53.037167 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:53.035610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:53.037167 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:53.035762    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:53.257501 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:53.257463    2583 generic.go:356] "Generic (PLEG): container finished" podID="931f933b-ce87-44d1-94ac-8dce917c3546" containerID="8ba6a2a69366d4fa6b9f1be85c640345964d625c193277fe8dec877189ca9fb3" exitCode=0
Sep 17 19:06:53.257674 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:53.257526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vp9lm" event={"ID":"931f933b-ce87-44d1-94ac-8dce917c3546","Type":"ContainerDied","Data":"8ba6a2a69366d4fa6b9f1be85c640345964d625c193277fe8dec877189ca9fb3"}
Sep 17 19:06:54.033149 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:54.033121    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:54.033287 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:54.033223    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:55.032945 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:55.032912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:55.033374 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:55.033036    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhl95" podUID="b5e92de3-419a-4bff-858a-d15f9509c77a"
Sep 17 19:06:55.033374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:55.033090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:55.033374 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:55.033183    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdcz8" podUID="abc81dc4-5b90-4a46-ad49-35d0fb866226"
Sep 17 19:06:56.034084 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.033898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:56.034523 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:56.034205    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g886c" podUID="5da89ee3-23c3-4a18-a046-0dde553add07"
Sep 17 19:06:56.158348 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.158325    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeReady"
Sep 17 19:06:56.158518 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.158471    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 19:06:56.229843 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.229801    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sn6l5"]
Sep 17 19:06:56.250040 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.250015    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9jvm6"]
Sep 17 19:06:56.250199 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.250158    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sn6l5"
Sep 17 19:06:56.256988 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.256926    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 19:06:56.256988 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.256926    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 19:06:56.257171 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.256990    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5sk68\""
Sep 17 19:06:56.257541 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.257523    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 19:06:56.263293 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.263271    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sn6l5"]
Sep 17 19:06:56.263400 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.263380    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.266314 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.266293    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zdmr2\""
Sep 17 19:06:56.266426 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.266411    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 19:06:56.266694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.266677    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 19:06:56.273522 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.273494    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jvm6"]
Sep 17 19:06:56.274439 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.274418    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xf97s"]
Sep 17 19:06:56.290448 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.290428    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.295256 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.295231    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 19:06:56.296476 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.296454    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xf97s"]
Sep 17 19:06:56.296637 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.296607    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xbtpt\""
Sep 17 19:06:56.296869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.296849    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 19:06:56.297072 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.297057    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 19:06:56.297338 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.297275    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 19:06:56.339638 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.339587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkrb\" (UniqueName: \"kubernetes.io/projected/5cc5e562-8481-4944-a52b-e0e7f6e5b991-kube-api-access-dzkrb\") pod \"ingress-canary-sn6l5\" (UID: \"5cc5e562-8481-4944-a52b-e0e7f6e5b991\") " pod="openshift-ingress-canary/ingress-canary-sn6l5"
Sep 17 19:06:56.339803 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.339654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jl7f\" (UniqueName: \"kubernetes.io/projected/63c7b622-334d-451f-9a47-f3a10c8d7f9b-kube-api-access-9jl7f\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.339803 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.339724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7b622-334d-451f-9a47-f3a10c8d7f9b-config-volume\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.339803 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.339798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63c7b622-334d-451f-9a47-f3a10c8d7f9b-tmp-dir\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.339927 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.339836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cc5e562-8481-4944-a52b-e0e7f6e5b991-cert\") pod \"ingress-canary-sn6l5\" (UID: \"5cc5e562-8481-4944-a52b-e0e7f6e5b991\") " pod="openshift-ingress-canary/ingress-canary-sn6l5"
Sep 17 19:06:56.339927 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.339852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63c7b622-334d-451f-9a47-f3a10c8d7f9b-metrics-tls\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.441205 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441123    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5525c384-23ac-4ae2-8b01-35a234b1dee8-crio-socket\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.441205 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63c7b622-334d-451f-9a47-f3a10c8d7f9b-tmp-dir\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.441444 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5525c384-23ac-4ae2-8b01-35a234b1dee8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.441444 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cc5e562-8481-4944-a52b-e0e7f6e5b991-cert\") pod \"ingress-canary-sn6l5\" (UID: \"5cc5e562-8481-4944-a52b-e0e7f6e5b991\") " pod="openshift-ingress-canary/ingress-canary-sn6l5"
Sep 17 19:06:56.441444 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63c7b622-334d-451f-9a47-f3a10c8d7f9b-metrics-tls\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.441444 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441404    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg5sb\" (UniqueName: \"kubernetes.io/projected/5525c384-23ac-4ae2-8b01-35a234b1dee8-kube-api-access-rg5sb\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.441656 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkrb\" (UniqueName: \"kubernetes.io/projected/5cc5e562-8481-4944-a52b-e0e7f6e5b991-kube-api-access-dzkrb\") pod \"ingress-canary-sn6l5\" (UID: \"5cc5e562-8481-4944-a52b-e0e7f6e5b991\") " pod="openshift-ingress-canary/ingress-canary-sn6l5"
Sep 17 19:06:56.441656 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5525c384-23ac-4ae2-8b01-35a234b1dee8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.441656 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441519    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63c7b622-334d-451f-9a47-f3a10c8d7f9b-tmp-dir\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.441656 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jl7f\" (UniqueName: \"kubernetes.io/projected/63c7b622-334d-451f-9a47-f3a10c8d7f9b-kube-api-access-9jl7f\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.441656 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7b622-334d-451f-9a47-f3a10c8d7f9b-config-volume\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.441656 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.441603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5525c384-23ac-4ae2-8b01-35a234b1dee8-data-volume\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.442160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.442133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63c7b622-334d-451f-9a47-f3a10c8d7f9b-config-volume\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.446072 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.446049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63c7b622-334d-451f-9a47-f3a10c8d7f9b-metrics-tls\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.446218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.446091    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cc5e562-8481-4944-a52b-e0e7f6e5b991-cert\") pod \"ingress-canary-sn6l5\" (UID: \"5cc5e562-8481-4944-a52b-e0e7f6e5b991\") " pod="openshift-ingress-canary/ingress-canary-sn6l5"
Sep 17 19:06:56.460097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.460075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jl7f\" (UniqueName: \"kubernetes.io/projected/63c7b622-334d-451f-9a47-f3a10c8d7f9b-kube-api-access-9jl7f\") pod \"dns-default-9jvm6\" (UID: \"63c7b622-334d-451f-9a47-f3a10c8d7f9b\") " pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.460343 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.460320    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkrb\" (UniqueName: \"kubernetes.io/projected/5cc5e562-8481-4944-a52b-e0e7f6e5b991-kube-api-access-dzkrb\") pod \"ingress-canary-sn6l5\" (UID: \"5cc5e562-8481-4944-a52b-e0e7f6e5b991\") " pod="openshift-ingress-canary/ingress-canary-sn6l5"
Sep 17 19:06:56.542876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.542837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg5sb\" (UniqueName: \"kubernetes.io/projected/5525c384-23ac-4ae2-8b01-35a234b1dee8-kube-api-access-rg5sb\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.543100 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.542901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5525c384-23ac-4ae2-8b01-35a234b1dee8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.543100 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.542947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5525c384-23ac-4ae2-8b01-35a234b1dee8-data-volume\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.543100 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.542980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5525c384-23ac-4ae2-8b01-35a234b1dee8-crio-socket\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.543100 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.543012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5525c384-23ac-4ae2-8b01-35a234b1dee8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.543526 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.543495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5525c384-23ac-4ae2-8b01-35a234b1dee8-data-volume\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.543673 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.543583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5525c384-23ac-4ae2-8b01-35a234b1dee8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.543740 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.543665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5525c384-23ac-4ae2-8b01-35a234b1dee8-crio-socket\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.545858 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.545838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5525c384-23ac-4ae2-8b01-35a234b1dee8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.553390 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.553369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg5sb\" (UniqueName: \"kubernetes.io/projected/5525c384-23ac-4ae2-8b01-35a234b1dee8-kube-api-access-rg5sb\") pod \"insights-runtime-extractor-xf97s\" (UID: \"5525c384-23ac-4ae2-8b01-35a234b1dee8\") " pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.564211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.564186    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sn6l5"
Sep 17 19:06:56.574558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.574522    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:06:56.600466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.600441    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xf97s"
Sep 17 19:06:56.745652 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.745555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:56.745652 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.745631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:56.745900 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:56.745750    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:56.745900 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:56.745837    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs podName:5da89ee3-23c3-4a18-a046-0dde553add07 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:28.745816323 +0000 UTC m=+67.873557725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs") pod "network-metrics-daemon-g886c" (UID: "5da89ee3-23c3-4a18-a046-0dde553add07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:56.745900 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:56.745750    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:56.746032 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:56.745917    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret podName:b5e92de3-419a-4bff-858a-d15f9509c77a nodeName:}" failed. No retries permitted until 2026-09-17 19:07:28.745895196 +0000 UTC m=+67.873636592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret") pod "global-pull-secret-syncer-xhl95" (UID: "b5e92de3-419a-4bff-858a-d15f9509c77a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:56.846861 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:56.846826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:56.847102 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:56.846970    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:56.847102 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:56.846988    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:56.847102 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:56.846996    2583 projected.go:196] Error preparing data for projected volume kube-api-access-vvd62 for pod openshift-network-diagnostics/network-check-target-vdcz8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:56.847102 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:06:56.847044    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62 podName:abc81dc4-5b90-4a46-ad49-35d0fb866226 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:28.847031478 +0000 UTC m=+67.974772857 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vvd62" (UniqueName: "kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62") pod "network-check-target-vdcz8" (UID: "abc81dc4-5b90-4a46-ad49-35d0fb866226") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:57.033546 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:57.033470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:06:57.033822 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:57.033485    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:06:57.036565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:57.036263    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:06:57.036565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:57.036269    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:06:57.036565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:57.036329    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jnl2p\""
Sep 17 19:06:57.036565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:57.036335    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:06:58.033510 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.033482    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:06:58.037107 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.037088    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:06:58.037808 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.037791    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sjq9g\""
Sep 17 19:06:58.071593 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.071455    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xf97s"]
Sep 17 19:06:58.074814 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.074791    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jvm6"]
Sep 17 19:06:58.075338 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.075319    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sn6l5"]
Sep 17 19:06:58.085310 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:58.085285    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc5e562_8481_4944_a52b_e0e7f6e5b991.slice/crio-40c8448faa4db0958dfc8c43dbdd7ea0397d70855e1b08401695b07648285e58 WatchSource:0}: Error finding container 40c8448faa4db0958dfc8c43dbdd7ea0397d70855e1b08401695b07648285e58: Status 404 returned error can't find the container with id 40c8448faa4db0958dfc8c43dbdd7ea0397d70855e1b08401695b07648285e58
Sep 17 19:06:58.087524 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:06:58.087190    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c7b622_334d_451f_9a47_f3a10c8d7f9b.slice/crio-ba399df0e60a09ffe9af151457824f04a3cc4e5676e722b381e8bbacd748dc0d WatchSource:0}: Error finding container ba399df0e60a09ffe9af151457824f04a3cc4e5676e722b381e8bbacd748dc0d: Status 404 returned error can't find the container with id ba399df0e60a09ffe9af151457824f04a3cc4e5676e722b381e8bbacd748dc0d
Sep 17 19:06:58.270902 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.270801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jvm6" event={"ID":"63c7b622-334d-451f-9a47-f3a10c8d7f9b","Type":"ContainerStarted","Data":"ba399df0e60a09ffe9af151457824f04a3cc4e5676e722b381e8bbacd748dc0d"}
Sep 17 19:06:58.271957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.271923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sn6l5" event={"ID":"5cc5e562-8481-4944-a52b-e0e7f6e5b991","Type":"ContainerStarted","Data":"40c8448faa4db0958dfc8c43dbdd7ea0397d70855e1b08401695b07648285e58"}
Sep 17 19:06:58.273374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.273349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xf97s" event={"ID":"5525c384-23ac-4ae2-8b01-35a234b1dee8","Type":"ContainerStarted","Data":"54aef8e690359ee4c6dc1b8d17b792709e4b572ef98b9498315a57ea68d1eb31"}
Sep 17 19:06:58.273479 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.273403    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xf97s" event={"ID":"5525c384-23ac-4ae2-8b01-35a234b1dee8","Type":"ContainerStarted","Data":"f3b3cde60c25ab257a52d763928c0354bc6250e788bfefd7e23147bfc7c69f80"}
Sep 17 19:06:58.275878 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.275856    2583 generic.go:356] "Generic (PLEG): container finished" podID="931f933b-ce87-44d1-94ac-8dce917c3546" containerID="f2a5b4eab656d32309d19933a7de01df0cd674ea7dd5391555f8222e420ba738" exitCode=0
Sep 17 19:06:58.275955 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:58.275890    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vp9lm" event={"ID":"931f933b-ce87-44d1-94ac-8dce917c3546","Type":"ContainerDied","Data":"f2a5b4eab656d32309d19933a7de01df0cd674ea7dd5391555f8222e420ba738"}
Sep 17 19:06:59.281202 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:59.281110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xf97s" event={"ID":"5525c384-23ac-4ae2-8b01-35a234b1dee8","Type":"ContainerStarted","Data":"fe7ddb25b7d4745493b355cacadb812ffcda0cf0965b223e2d3b91e793967c16"}
Sep 17 19:06:59.285510 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:59.285478    2583 generic.go:356] "Generic (PLEG): container finished" podID="931f933b-ce87-44d1-94ac-8dce917c3546" containerID="f21f0cc7dd219002747f8d5559bc05232b6ba605fd95ee36e18a89ab4c06c623" exitCode=0
Sep 17 19:06:59.285673 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:06:59.285533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vp9lm" event={"ID":"931f933b-ce87-44d1-94ac-8dce917c3546","Type":"ContainerDied","Data":"f21f0cc7dd219002747f8d5559bc05232b6ba605fd95ee36e18a89ab4c06c623"}
Sep 17 19:07:00.294473 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.294395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vp9lm" event={"ID":"931f933b-ce87-44d1-94ac-8dce917c3546","Type":"ContainerStarted","Data":"74968e98ead73f5508a1d68bea9b96754a9f23684c7a0a01226d97111d5fb044"}
Sep 17 19:07:00.323541 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.323365    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vp9lm" podStartSLOduration=8.501199029 podStartE2EDuration="37.323346235s" totalImagesPullingTime="28.822147206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:00.321481557 +0000 UTC m=+39.449222995" watchObservedRunningTime="2026-09-17 19:07:00.323346235 +0000 UTC m=+39.451087637"
Sep 17 19:07:00.337859 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.337830    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"]
Sep 17 19:07:00.339894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.339876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.344367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.344189    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9d9pw\""
Sep 17 19:07:00.344367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.344200    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 19:07:00.344367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.344232    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 19:07:00.344367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.344321    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 19:07:00.344712 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.344385    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 19:07:00.344712 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.344461    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 19:07:00.355606 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.355586    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"]
Sep 17 19:07:00.476814 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.476779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.476984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.476909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.476984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.476940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.476984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.476969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzt8n\" (UniqueName: \"kubernetes.io/projected/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-kube-api-access-tzt8n\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.578184 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.578096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.578184 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.578177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.578380 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:07:00.578276    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 19:07:00.578380 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:07:00.578329    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-prometheus-operator-tls podName:b5c9c5f4-869b-4b14-8f1c-1ab03e01b013 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:01.078314508 +0000 UTC m=+40.206055891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-4mf42" (UID: "b5c9c5f4-869b-4b14-8f1c-1ab03e01b013") : secret "prometheus-operator-tls" not found
Sep 17 19:07:00.578630 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.578587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.578737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.578662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzt8n\" (UniqueName: \"kubernetes.io/projected/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-kube-api-access-tzt8n\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.578966 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.578942    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.582691 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.582661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:00.598741 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:00.598684    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzt8n\" (UniqueName: \"kubernetes.io/projected/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-kube-api-access-tzt8n\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:01.082457 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:01.082409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:01.085952 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:01.085926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5c9c5f4-869b-4b14-8f1c-1ab03e01b013-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4mf42\" (UID: \"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:01.253476 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:01.253447    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"
Sep 17 19:07:01.303042 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:01.302990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jvm6" event={"ID":"63c7b622-334d-451f-9a47-f3a10c8d7f9b","Type":"ContainerStarted","Data":"220a7afbe76a77d9d7baddb9891d3957e7ad3fe5fa4b59e2c71e8e7484a1e46f"}
Sep 17 19:07:01.306452 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:01.306418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sn6l5" event={"ID":"5cc5e562-8481-4944-a52b-e0e7f6e5b991","Type":"ContainerStarted","Data":"d4e65775afdb9771177def1693a41ad847a753fae1bbaa2ec6c3be9892ac5730"}
Sep 17 19:07:01.310417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:01.310385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xf97s" event={"ID":"5525c384-23ac-4ae2-8b01-35a234b1dee8","Type":"ContainerStarted","Data":"d4d952de782e71d1e22277e5f798e55c052709e3b5aebbd1560d2e7be20fc613"}
Sep 17 19:07:01.337525 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:01.337202    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sn6l5" podStartSLOduration=2.246644953 podStartE2EDuration="5.337178348s" totalImagesPullingTime="3.090533395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:01.33508159 +0000 UTC m=+40.462822996" watchObservedRunningTime="2026-09-17 19:07:01.337178348 +0000 UTC m=+40.464919750"
Sep 17 19:07:01.394129 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:01.394048    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xf97s" podStartSLOduration=2.496326846 podStartE2EDuration="5.394028438s" totalImagesPullingTime="2.897701592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:01.393385957 +0000 UTC m=+40.521127359" watchObservedRunningTime="2026-09-17 19:07:01.394028438 +0000 UTC m=+40.521769842"
Sep 17 19:07:01.543890 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:01.543854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42"]
Sep 17 19:07:01.548655 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:01.548602    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c9c5f4_869b_4b14_8f1c_1ab03e01b013.slice/crio-aaf92810c20ab5d216d6d3733502f043084f98207207d8159274d209426b103b WatchSource:0}: Error finding container aaf92810c20ab5d216d6d3733502f043084f98207207d8159274d209426b103b: Status 404 returned error can't find the container with id aaf92810c20ab5d216d6d3733502f043084f98207207d8159274d209426b103b
Sep 17 19:07:02.315422 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:02.315382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42" event={"ID":"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013","Type":"ContainerStarted","Data":"aaf92810c20ab5d216d6d3733502f043084f98207207d8159274d209426b103b"}
Sep 17 19:07:02.317717 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:02.317654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jvm6" event={"ID":"63c7b622-334d-451f-9a47-f3a10c8d7f9b","Type":"ContainerStarted","Data":"9ed2f178df379990d87a5b0f624ca2c13d2c31aa1d228a9ebc410cdf3ef0f875"}
Sep 17 19:07:02.318034 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:02.318011    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:07:03.322867 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:03.322785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42" event={"ID":"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013","Type":"ContainerStarted","Data":"288c6c6b6a082f22a348716edd5f4496900acbce52c78d7def9ebac1ef5b5caa"}
Sep 17 19:07:03.322867 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:03.322845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42" event={"ID":"b5c9c5f4-869b-4b14-8f1c-1ab03e01b013","Type":"ContainerStarted","Data":"5ecc2d1f44114284b8f6700826dce0dcf0bacad4fd7453172554748720281fa7"}
Sep 17 19:07:03.341103 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:03.341058    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9jvm6" podStartSLOduration=4.257781287 podStartE2EDuration="7.341042596s" totalImagesPullingTime="3.083261309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:02.343438692 +0000 UTC m=+41.471180092" watchObservedRunningTime="2026-09-17 19:07:03.341042596 +0000 UTC m=+42.468784041"
Sep 17 19:07:05.746555 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.746502    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4mf42" podStartSLOduration=4.342445027 podStartE2EDuration="5.746484547s" totalImagesPullingTime="1.40403952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:03.340674264 +0000 UTC m=+42.468415662" watchObservedRunningTime="2026-09-17 19:07:05.746484547 +0000 UTC m=+44.874225947"
Sep 17 19:07:05.747004 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.746676    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"]
Sep 17 19:07:05.750038 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.750011    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:05.752840 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.752818    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 19:07:05.752973 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.752876    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 19:07:05.752973 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.752876    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mbbdp\""
Sep 17 19:07:05.762281 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.762257    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"]
Sep 17 19:07:05.779934 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.779909    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"]
Sep 17 19:07:05.781825 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.781810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:05.785183 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.785162    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 19:07:05.785287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.785171    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 19:07:05.785572 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.785550    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r6849\""
Sep 17 19:07:05.785761 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.785748    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 19:07:05.793916 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.793898    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qhp6v"]
Sep 17 19:07:05.795837 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.795823    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.799317 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.799269    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 19:07:05.799425 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.799414    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 19:07:05.799649 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.799606    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6bgp8\""
Sep 17 19:07:05.800233 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.800215    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 19:07:05.806902 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.806880    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"]
Sep 17 19:07:05.922268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:05.922437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-wtmp\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.922437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgfp\" (UniqueName: \"kubernetes.io/projected/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-api-access-khgfp\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:05.922437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.922437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:05.922437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-accelerators-collector-config\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.922437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-textfile\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.922437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:05.922694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6541cbbf-c6d5-49b1-9637-dfa61a479d57-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:05.922694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-tls\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.922694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922516    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkmk9\" (UniqueName: \"kubernetes.io/projected/264b1a88-c09c-466e-b0c1-b5878a8c0184-kube-api-access-zkmk9\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.922694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/264b1a88-c09c-466e-b0c1-b5878a8c0184-root\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.922694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922582    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69e66a75-f4d6-4f86-ad91-31f4aebd0309-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:05.922694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922643    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/264b1a88-c09c-466e-b0c1-b5878a8c0184-sys\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.922694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69e66a75-f4d6-4f86-ad91-31f4aebd0309-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:05.922694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx2pc\" (UniqueName: \"kubernetes.io/projected/69e66a75-f4d6-4f86-ad91-31f4aebd0309-kube-api-access-lx2pc\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:05.922951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6541cbbf-c6d5-49b1-9637-dfa61a479d57-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:05.922951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264b1a88-c09c-466e-b0c1-b5878a8c0184-metrics-client-ca\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:05.922951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:05.922798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69e66a75-f4d6-4f86-ad91-31f4aebd0309-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.023790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.023790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-wtmp\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.023790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgfp\" (UniqueName: \"kubernetes.io/projected/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-api-access-khgfp\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.023790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-accelerators-collector-config\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-textfile\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6541cbbf-c6d5-49b1-9637-dfa61a479d57-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023909    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-wtmp\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-tls\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkmk9\" (UniqueName: \"kubernetes.io/projected/264b1a88-c09c-466e-b0c1-b5878a8c0184-kube-api-access-zkmk9\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.023975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/264b1a88-c09c-466e-b0c1-b5878a8c0184-root\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024000    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69e66a75-f4d6-4f86-ad91-31f4aebd0309-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:07:06.024010    2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/264b1a88-c09c-466e-b0c1-b5878a8c0184-sys\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:07:06.024068    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-tls podName:6541cbbf-c6d5-49b1-9637-dfa61a479d57 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:06.524049382 +0000 UTC m=+45.651790775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-zqfxg" (UID: "6541cbbf-c6d5-49b1-9637-dfa61a479d57") : secret "kube-state-metrics-tls" not found
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/264b1a88-c09c-466e-b0c1-b5878a8c0184-sys\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69e66a75-f4d6-4f86-ad91-31f4aebd0309-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.024854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2pc\" (UniqueName: \"kubernetes.io/projected/69e66a75-f4d6-4f86-ad91-31f4aebd0309-kube-api-access-lx2pc\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.024854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6541cbbf-c6d5-49b1-9637-dfa61a479d57-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.024854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264b1a88-c09c-466e-b0c1-b5878a8c0184-metrics-client-ca\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.024854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69e66a75-f4d6-4f86-ad91-31f4aebd0309-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.024854 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:07:06.024374    2583 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 19:07:06.024854 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:07:06.024420    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e66a75-f4d6-4f86-ad91-31f4aebd0309-openshift-state-metrics-tls podName:69e66a75-f4d6-4f86-ad91-31f4aebd0309 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:06.524404558 +0000 UTC m=+45.652145951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/69e66a75-f4d6-4f86-ad91-31f4aebd0309-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-n9q67" (UID: "69e66a75-f4d6-4f86-ad91-31f4aebd0309") : secret "openshift-state-metrics-tls" not found
Sep 17 19:07:06.024854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024536    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-accelerators-collector-config\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.025195 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6541cbbf-c6d5-49b1-9637-dfa61a479d57-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.025195 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024835    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.025195 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/264b1a88-c09c-466e-b0c1-b5878a8c0184-root\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.025195 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.024965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6541cbbf-c6d5-49b1-9637-dfa61a479d57-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.025195 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.025059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69e66a75-f4d6-4f86-ad91-31f4aebd0309-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.025464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.025287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264b1a88-c09c-466e-b0c1-b5878a8c0184-metrics-client-ca\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.025464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.025357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-textfile\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.026607 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.026575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.027126 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.027100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69e66a75-f4d6-4f86-ad91-31f4aebd0309-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.027199 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.027114    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-tls\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.027273 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.027256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/264b1a88-c09c-466e-b0c1-b5878a8c0184-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.034476 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.034455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgfp\" (UniqueName: \"kubernetes.io/projected/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-api-access-khgfp\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.035271 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.035250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2pc\" (UniqueName: \"kubernetes.io/projected/69e66a75-f4d6-4f86-ad91-31f4aebd0309-kube-api-access-lx2pc\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.036545 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.036530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkmk9\" (UniqueName: \"kubernetes.io/projected/264b1a88-c09c-466e-b0c1-b5878a8c0184-kube-api-access-zkmk9\") pod \"node-exporter-qhp6v\" (UID: \"264b1a88-c09c-466e-b0c1-b5878a8c0184\") " pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.104502 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.104475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qhp6v"
Sep 17 19:07:06.114394 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:06.114368    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264b1a88_c09c_466e_b0c1_b5878a8c0184.slice/crio-fae7fbf314f2be451c732e9ea92fd186391a67c1084dd3b0acd7a436b4ca41df WatchSource:0}: Error finding container fae7fbf314f2be451c732e9ea92fd186391a67c1084dd3b0acd7a436b4ca41df: Status 404 returned error can't find the container with id fae7fbf314f2be451c732e9ea92fd186391a67c1084dd3b0acd7a436b4ca41df
Sep 17 19:07:06.333811 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.333726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qhp6v" event={"ID":"264b1a88-c09c-466e-b0c1-b5878a8c0184","Type":"ContainerStarted","Data":"fae7fbf314f2be451c732e9ea92fd186391a67c1084dd3b0acd7a436b4ca41df"}
Sep 17 19:07:06.527410 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.527360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69e66a75-f4d6-4f86-ad91-31f4aebd0309-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.527574 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.527445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.530018 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.529993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69e66a75-f4d6-4f86-ad91-31f4aebd0309-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-n9q67\" (UID: \"69e66a75-f4d6-4f86-ad91-31f4aebd0309\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.530123 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.530049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6541cbbf-c6d5-49b1-9637-dfa61a479d57-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-zqfxg\" (UID: \"6541cbbf-c6d5-49b1-9637-dfa61a479d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.659252 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.659160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"
Sep 17 19:07:06.690762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.690728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"
Sep 17 19:07:06.831910 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.831879    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:07:06.834595 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.834572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.837079 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.837025    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 19:07:06.837410 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.837386    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 19:07:06.839377 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.837386    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 19:07:06.839761 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.839428    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 19:07:06.839761 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.839512    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 19:07:06.839761 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.837587    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 19:07:06.839761 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.837572    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 19:07:06.839761 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.837767    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 19:07:06.839761 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.837654    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 19:07:06.840203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.839920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xmt25\""
Sep 17 19:07:06.856264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.856229    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:07:06.930372 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930372 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w82vb\" (UniqueName: \"kubernetes.io/projected/33731d5a-c3dc-4464-ba64-686c33118bd9-kube-api-access-w82vb\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930417    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-config-volume\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33731d5a-c3dc-4464-ba64-686c33118bd9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-web-config\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/33731d5a-c3dc-4464-ba64-686c33118bd9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33731d5a-c3dc-4464-ba64-686c33118bd9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33731d5a-c3dc-4464-ba64-686c33118bd9-config-out\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.930877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.930731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33731d5a-c3dc-4464-ba64-686c33118bd9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:06.935152 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.935123    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67"]
Sep 17 19:07:06.939786 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:06.939761    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e66a75_f4d6_4f86_ad91_31f4aebd0309.slice/crio-e9b6acc18463b351ea568dca3cdbde445c466b22c50130937e665a7910bc96c6 WatchSource:0}: Error finding container e9b6acc18463b351ea568dca3cdbde445c466b22c50130937e665a7910bc96c6: Status 404 returned error can't find the container with id e9b6acc18463b351ea568dca3cdbde445c466b22c50130937e665a7910bc96c6
Sep 17 19:07:06.978197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:06.978168    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-zqfxg"]
Sep 17 19:07:06.982496 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:06.982468    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6541cbbf_c6d5_49b1_9637_dfa61a479d57.slice/crio-6806ed29ec40f13f5f542a1964ecc1eb759bb1d785f6d15eda1235ab3f2ed9d0 WatchSource:0}: Error finding container 6806ed29ec40f13f5f542a1964ecc1eb759bb1d785f6d15eda1235ab3f2ed9d0: Status 404 returned error can't find the container with id 6806ed29ec40f13f5f542a1964ecc1eb759bb1d785f6d15eda1235ab3f2ed9d0
Sep 17 19:07:07.031933 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.031905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.031959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.031988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33731d5a-c3dc-4464-ba64-686c33118bd9-config-out\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33731d5a-c3dc-4464-ba64-686c33118bd9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032107    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w82vb\" (UniqueName: \"kubernetes.io/projected/33731d5a-c3dc-4464-ba64-686c33118bd9-kube-api-access-w82vb\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-config-volume\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33731d5a-c3dc-4464-ba64-686c33118bd9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-web-config\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032572 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:07:07.032267    2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 19:07:07.032572 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/33731d5a-c3dc-4464-ba64-686c33118bd9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032572 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.032317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33731d5a-c3dc-4464-ba64-686c33118bd9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.032572 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:07:07.032346    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-main-tls podName:33731d5a-c3dc-4464-ba64-686c33118bd9 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:07.532324089 +0000 UTC m=+46.660065492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "33731d5a-c3dc-4464-ba64-686c33118bd9") : secret "alertmanager-main-tls" not found
Sep 17 19:07:07.033514 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:07:07.033422    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33731d5a-c3dc-4464-ba64-686c33118bd9-alertmanager-trusted-ca-bundle podName:33731d5a-c3dc-4464-ba64-686c33118bd9 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:07.533403412 +0000 UTC m=+46.661144814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/33731d5a-c3dc-4464-ba64-686c33118bd9-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "33731d5a-c3dc-4464-ba64-686c33118bd9") : configmap references non-existent config key: ca-bundle.crt
Sep 17 19:07:07.034077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.033958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/33731d5a-c3dc-4464-ba64-686c33118bd9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.035173 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.035118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33731d5a-c3dc-4464-ba64-686c33118bd9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.036218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.036168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33731d5a-c3dc-4464-ba64-686c33118bd9-config-out\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.036304 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.036260    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.036514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.036489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.036638 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.036563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-web-config\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.037027 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.036986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.037940 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.037901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-config-volume\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.038580 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.038556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.038699 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.038676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33731d5a-c3dc-4464-ba64-686c33118bd9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.041966 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.041944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w82vb\" (UniqueName: \"kubernetes.io/projected/33731d5a-c3dc-4464-ba64-686c33118bd9-kube-api-access-w82vb\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.339685 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.339577    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67" event={"ID":"69e66a75-f4d6-4f86-ad91-31f4aebd0309","Type":"ContainerStarted","Data":"dda58028a543b28745b9ae22d8c3a3b63da5d4dc68e505c7a04f1d56219412b5"}
Sep 17 19:07:07.339685 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.339673    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67" event={"ID":"69e66a75-f4d6-4f86-ad91-31f4aebd0309","Type":"ContainerStarted","Data":"75733b43a06c7d12acd4bfc46191dea96822cee9704cf6ca104ed03e8e0f2905"}
Sep 17 19:07:07.339882 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.339704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67" event={"ID":"69e66a75-f4d6-4f86-ad91-31f4aebd0309","Type":"ContainerStarted","Data":"e9b6acc18463b351ea568dca3cdbde445c466b22c50130937e665a7910bc96c6"}
Sep 17 19:07:07.341029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.341004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg" event={"ID":"6541cbbf-c6d5-49b1-9637-dfa61a479d57","Type":"ContainerStarted","Data":"6806ed29ec40f13f5f542a1964ecc1eb759bb1d785f6d15eda1235ab3f2ed9d0"}
Sep 17 19:07:07.536757 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.536724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.537090 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.536838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33731d5a-c3dc-4464-ba64-686c33118bd9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.537594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.537567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33731d5a-c3dc-4464-ba64-686c33118bd9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.540054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.540030    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/33731d5a-c3dc-4464-ba64-686c33118bd9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"33731d5a-c3dc-4464-ba64-686c33118bd9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.746218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.746182    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:07.778092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.778061    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5677f75b99-pz5zs"]
Sep 17 19:07:07.781647 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.781178    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.785751 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.785724    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 19:07:07.785751 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.785748    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cupps4ndobon9\""
Sep 17 19:07:07.786138 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.785958    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 19:07:07.786138 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.786004    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 19:07:07.786138 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.786039    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-z89mz\""
Sep 17 19:07:07.786138 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.786044    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 19:07:07.786439 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.786248    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 19:07:07.803955 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.803922    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5677f75b99-pz5zs"]
Sep 17 19:07:07.840546 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.840511    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-tls\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.840980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.840560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvmr\" (UniqueName: \"kubernetes.io/projected/9c6ceaf7-95b8-41f7-812e-8f93c1636316-kube-api-access-zlvmr\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.840980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.840648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.840980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.840725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c6ceaf7-95b8-41f7-812e-8f93c1636316-metrics-client-ca\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.840980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.840780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-grpc-tls\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.840980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.840830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.840980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.840874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.840980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.840926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.942230 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.942198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c6ceaf7-95b8-41f7-812e-8f93c1636316-metrics-client-ca\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.942365 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.942245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-grpc-tls\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.942365 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.942292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.942365 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.942329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.942564 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.942545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.942670 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.942607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-tls\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.942670 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.942661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvmr\" (UniqueName: \"kubernetes.io/projected/9c6ceaf7-95b8-41f7-812e-8f93c1636316-kube-api-access-zlvmr\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.942779 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.942705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.943058 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.943029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c6ceaf7-95b8-41f7-812e-8f93c1636316-metrics-client-ca\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.945348 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.945322    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.945435 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.945380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-tls\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.945634 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.945589    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.945700 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.945608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-grpc-tls\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.945700 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.945599    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.945798 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.945719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9c6ceaf7-95b8-41f7-812e-8f93c1636316-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:07.952095 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:07.952076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvmr\" (UniqueName: \"kubernetes.io/projected/9c6ceaf7-95b8-41f7-812e-8f93c1636316-kube-api-access-zlvmr\") pod \"thanos-querier-5677f75b99-pz5zs\" (UID: \"9c6ceaf7-95b8-41f7-812e-8f93c1636316\") " pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:08.041821 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:08.041751    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:07:08.096066 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:08.096032    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:08.228725 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:08.228689    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33731d5a_c3dc_4464_ba64_686c33118bd9.slice/crio-99e522849c10fc0f622f28c80980c374604513c7152c2040e369c1b0321dae47 WatchSource:0}: Error finding container 99e522849c10fc0f622f28c80980c374604513c7152c2040e369c1b0321dae47: Status 404 returned error can't find the container with id 99e522849c10fc0f622f28c80980c374604513c7152c2040e369c1b0321dae47
Sep 17 19:07:08.347304 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:08.347208    2583 generic.go:356] "Generic (PLEG): container finished" podID="264b1a88-c09c-466e-b0c1-b5878a8c0184" containerID="3acb3642fc28f6b2753237aa92a07a180e5b37ec4f8f1bcc70acc9399e3c6d8d" exitCode=0
Sep 17 19:07:08.347442 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:08.347363    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qhp6v" event={"ID":"264b1a88-c09c-466e-b0c1-b5878a8c0184","Type":"ContainerDied","Data":"3acb3642fc28f6b2753237aa92a07a180e5b37ec4f8f1bcc70acc9399e3c6d8d"}
Sep 17 19:07:08.349046 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:08.349020    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"33731d5a-c3dc-4464-ba64-686c33118bd9","Type":"ContainerStarted","Data":"99e522849c10fc0f622f28c80980c374604513c7152c2040e369c1b0321dae47"}
Sep 17 19:07:08.819424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:08.819392    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5677f75b99-pz5zs"]
Sep 17 19:07:08.824096 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:08.824070    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6ceaf7_95b8_41f7_812e_8f93c1636316.slice/crio-af530f25aa580603ddd1ea810a9c1b6c0cd61dc5c8f1b8c869d1d12c707de7ce WatchSource:0}: Error finding container af530f25aa580603ddd1ea810a9c1b6c0cd61dc5c8f1b8c869d1d12c707de7ce: Status 404 returned error can't find the container with id af530f25aa580603ddd1ea810a9c1b6c0cd61dc5c8f1b8c869d1d12c707de7ce
Sep 17 19:07:09.355027 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.354988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs" event={"ID":"9c6ceaf7-95b8-41f7-812e-8f93c1636316","Type":"ContainerStarted","Data":"af530f25aa580603ddd1ea810a9c1b6c0cd61dc5c8f1b8c869d1d12c707de7ce"}
Sep 17 19:07:09.357850 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.357818    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qhp6v" event={"ID":"264b1a88-c09c-466e-b0c1-b5878a8c0184","Type":"ContainerStarted","Data":"7fd11030511069901109da7b337f52c35199f2c8284d860a0daac8909b3bb0a1"}
Sep 17 19:07:09.357989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.357877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qhp6v" event={"ID":"264b1a88-c09c-466e-b0c1-b5878a8c0184","Type":"ContainerStarted","Data":"872cf96fcef50bac3a31c9ea5d3e175338a800285ad64bc55402f666bc3dc3c6"}
Sep 17 19:07:09.360342 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.360313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67" event={"ID":"69e66a75-f4d6-4f86-ad91-31f4aebd0309","Type":"ContainerStarted","Data":"7209a3764146e61cbe2a55046aff68f77f0ef0fce6a260dce85f3ceea94dc87f"}
Sep 17 19:07:09.363130 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.362729    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg" event={"ID":"6541cbbf-c6d5-49b1-9637-dfa61a479d57","Type":"ContainerStarted","Data":"05bb156065ce2efd1c536583578c0f431cd195e84e8d2c6315637f341482a41b"}
Sep 17 19:07:09.363130 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.362780    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg" event={"ID":"6541cbbf-c6d5-49b1-9637-dfa61a479d57","Type":"ContainerStarted","Data":"4480c4ce6c0a4c4a2164b1a868b6167c3d994134d851cfbd4afd211051b48214"}
Sep 17 19:07:09.363130 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.362809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg" event={"ID":"6541cbbf-c6d5-49b1-9637-dfa61a479d57","Type":"ContainerStarted","Data":"2189303f31b49a5a641328f083dfded7ab9a10541fb93f184d5a4c1253756f38"}
Sep 17 19:07:09.379273 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.379231    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qhp6v" podStartSLOduration=3.026155992 podStartE2EDuration="4.379214897s" totalImagesPullingTime="1.276557814s" totalInitContainerRuntime="76.501091ms" isStatefulPod=true podCreationTimestamp="2026-09-17 19:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:09.378058337 +0000 UTC m=+48.505799739" watchObservedRunningTime="2026-09-17 19:07:09.379214897 +0000 UTC m=+48.506956300"
Sep 17 19:07:09.400118 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.400067    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-zqfxg" podStartSLOduration=2.902311851 podStartE2EDuration="4.400046074s" totalImagesPullingTime="1.497734223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:09.39925903 +0000 UTC m=+48.527000432" watchObservedRunningTime="2026-09-17 19:07:09.400046074 +0000 UTC m=+48.527787476"
Sep 17 19:07:09.424291 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:09.424246    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n9q67" podStartSLOduration=3.028886148 podStartE2EDuration="4.424227208s" totalImagesPullingTime="1.39534106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:09.423334574 +0000 UTC m=+48.551075977" watchObservedRunningTime="2026-09-17 19:07:09.424227208 +0000 UTC m=+48.551968611"
Sep 17 19:07:10.367417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.367383    2583 generic.go:356] "Generic (PLEG): container finished" podID="33731d5a-c3dc-4464-ba64-686c33118bd9" containerID="2dc2443ae56e134147feec2f0c58c101f8332d58fdf2d295dd56874a04a9fd1a" exitCode=0
Sep 17 19:07:10.367881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.367480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"33731d5a-c3dc-4464-ba64-686c33118bd9","Type":"ContainerDied","Data":"2dc2443ae56e134147feec2f0c58c101f8332d58fdf2d295dd56874a04a9fd1a"}
Sep 17 19:07:10.629401 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.629318    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"]
Sep 17 19:07:10.659556 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.659523    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"]
Sep 17 19:07:10.659708 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.659646    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:10.662407 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.662386    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lmrtc\""
Sep 17 19:07:10.663086 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.663066    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 19:07:10.768635 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.768579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxj2\" (UniqueName: \"kubernetes.io/projected/1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063-kube-api-access-jrxj2\") pod \"monitoring-plugin-5984459ff8-nwjbm\" (UID: \"1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:10.768798 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.768693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-nwjbm\" (UID: \"1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:10.869492 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.869425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxj2\" (UniqueName: \"kubernetes.io/projected/1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063-kube-api-access-jrxj2\") pod \"monitoring-plugin-5984459ff8-nwjbm\" (UID: \"1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:10.869695 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.869579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-nwjbm\" (UID: \"1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:10.872193 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.872167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-nwjbm\" (UID: \"1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:10.884083 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.884026    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxj2\" (UniqueName: \"kubernetes.io/projected/1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063-kube-api-access-jrxj2\") pod \"monitoring-plugin-5984459ff8-nwjbm\" (UID: \"1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:10.968820 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:10.968793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:11.254316 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:11.254238    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"]
Sep 17 19:07:11.808814 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:11.808778    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9df7ef_ed1d_48a9_a8e7_cdcd44ac6063.slice/crio-0b01f928ee88ffa31911f3186c3734ab5d97578036e6f8151f247b0384185ef1 WatchSource:0}: Error finding container 0b01f928ee88ffa31911f3186c3734ab5d97578036e6f8151f247b0384185ef1: Status 404 returned error can't find the container with id 0b01f928ee88ffa31911f3186c3734ab5d97578036e6f8151f247b0384185ef1
Sep 17 19:07:12.219832 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.219803    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 19:07:12.241157 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.241125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.244511 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.244490    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 19:07:12.244651 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.244490    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 19:07:12.245402 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.245377    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 19:07:12.245869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.245847    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6kgnd\""
Sep 17 19:07:12.248129 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.248095    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 19:07:12.248129 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.248122    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 19:07:12.248648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.248547    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 19:07:12.250947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.249181    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 19:07:12.250947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.249271    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ellnai2ch8rfo\""
Sep 17 19:07:12.250947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.249376    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 19:07:12.250947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.249498    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 19:07:12.250947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.249562    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 19:07:12.250947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.249748    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 19:07:12.250947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.249793    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 19:07:12.250947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.250011    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 19:07:12.251439 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.251419    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 19:07:12.254573 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.254536    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 19:07:12.255677 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.255656    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 19:07:12.283893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.283855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e9ca9fe-6839-4aaf-808e-4ba059036268-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.283893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.283885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.283906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-web-config\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.283937    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.283989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-config\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cv55\" (UniqueName: \"kubernetes.io/projected/9e9ca9fe-6839-4aaf-808e-4ba059036268-kube-api-access-7cv55\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e9ca9fe-6839-4aaf-808e-4ba059036268-config-out\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284289    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284403    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.284797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.284464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.325402 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.325369    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9jvm6"
Sep 17 19:07:12.377603 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.377565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm" event={"ID":"1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063","Type":"ContainerStarted","Data":"0b01f928ee88ffa31911f3186c3734ab5d97578036e6f8151f247b0384185ef1"}
Sep 17 19:07:12.384968 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.384926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385128 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e9ca9fe-6839-4aaf-808e-4ba059036268-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385128 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385242 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-web-config\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385242 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385351 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-config\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385351 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385442 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cv55\" (UniqueName: \"kubernetes.io/projected/9e9ca9fe-6839-4aaf-808e-4ba059036268-kube-api-access-7cv55\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385442 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385759 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e9ca9fe-6839-4aaf-808e-4ba059036268-config-out\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385759 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385759 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385759 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.385759 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.386160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.386160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.386160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.385831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.386160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.386131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.387146 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.387115    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.387482 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.387455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.388716 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.388344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.388716 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.388683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-config\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.389011 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.388989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e9ca9fe-6839-4aaf-808e-4ba059036268-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.389203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.389179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-web-config\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.389337 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.389316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.389438 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.389365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.389788 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.389764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.389880 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.389774    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e9ca9fe-6839-4aaf-808e-4ba059036268-config-out\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.391582 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.391538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.391815 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.391792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.392087 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.392062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.392567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.392537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.392812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.392793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.393138 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.393121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9e9ca9fe-6839-4aaf-808e-4ba059036268-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.393787 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.393766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9e9ca9fe-6839-4aaf-808e-4ba059036268-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.405057 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.405036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cv55\" (UniqueName: \"kubernetes.io/projected/9e9ca9fe-6839-4aaf-808e-4ba059036268-kube-api-access-7cv55\") pod \"prometheus-k8s-0\" (UID: \"9e9ca9fe-6839-4aaf-808e-4ba059036268\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.554294 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.554196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:12.993604 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:12.993581    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 19:07:12.999970 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:12.999898    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9ca9fe_6839_4aaf_808e_4ba059036268.slice/crio-8728dfb467e3690f0f623192cfb04fb5667ea6c89b3750b12435f0be6d5769a2 WatchSource:0}: Error finding container 8728dfb467e3690f0f623192cfb04fb5667ea6c89b3750b12435f0be6d5769a2: Status 404 returned error can't find the container with id 8728dfb467e3690f0f623192cfb04fb5667ea6c89b3750b12435f0be6d5769a2
Sep 17 19:07:13.383635 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.383532    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e9ca9fe-6839-4aaf-808e-4ba059036268" containerID="b57ce25995cb02d314270b594478e0181656bc4a3535adfbb15f5268497d1517" exitCode=0
Sep 17 19:07:13.383635 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.383580    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9ca9fe-6839-4aaf-808e-4ba059036268","Type":"ContainerDied","Data":"b57ce25995cb02d314270b594478e0181656bc4a3535adfbb15f5268497d1517"}
Sep 17 19:07:13.383805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.383677    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9ca9fe-6839-4aaf-808e-4ba059036268","Type":"ContainerStarted","Data":"8728dfb467e3690f0f623192cfb04fb5667ea6c89b3750b12435f0be6d5769a2"}
Sep 17 19:07:13.386371 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.386345    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs" event={"ID":"9c6ceaf7-95b8-41f7-812e-8f93c1636316","Type":"ContainerStarted","Data":"b9d9230bbf977d36d8036c195c53c1c3e26e594efe395724014412912ceaf3bc"}
Sep 17 19:07:13.386495 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.386420    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs" event={"ID":"9c6ceaf7-95b8-41f7-812e-8f93c1636316","Type":"ContainerStarted","Data":"69baf108209d0601f32ac4fe1448f365e0d2f1a9f815babbea7bf2e303802c70"}
Sep 17 19:07:13.386495 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.386465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs" event={"ID":"9c6ceaf7-95b8-41f7-812e-8f93c1636316","Type":"ContainerStarted","Data":"55c556bd4773c88a087486639061b2d461f8a37fbcd7a129cd6084f3e840dc35"}
Sep 17 19:07:13.390166 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.390136    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"33731d5a-c3dc-4464-ba64-686c33118bd9","Type":"ContainerStarted","Data":"686f69af49365a934d51d630192ca0b84d2e90847b8f3cb8f915eecb60905da3"}
Sep 17 19:07:13.390255 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.390205    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"33731d5a-c3dc-4464-ba64-686c33118bd9","Type":"ContainerStarted","Data":"47a44a551cfaf2f673b3fb3fb216c79e6f69714f3645f133aa994792ef0594ef"}
Sep 17 19:07:13.390255 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.390245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"33731d5a-c3dc-4464-ba64-686c33118bd9","Type":"ContainerStarted","Data":"d714a6bd5828a4b7549013e462a1ac44b4c241fbdf1282120a5b5f4ceddb4a64"}
Sep 17 19:07:13.390355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.390285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"33731d5a-c3dc-4464-ba64-686c33118bd9","Type":"ContainerStarted","Data":"534111eed158f167de192d324b3f4a4c03a846aeabdc1bad8dd72480afb2b0f8"}
Sep 17 19:07:13.390355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:13.390326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"33731d5a-c3dc-4464-ba64-686c33118bd9","Type":"ContainerStarted","Data":"75fa42df50e73605d0f9c4d0117cb628c199c476e10cc556ddd6ce0b9c1248d1"}
Sep 17 19:07:14.396518 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:14.396483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm" event={"ID":"1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063","Type":"ContainerStarted","Data":"6c37cec41d95ed590bc331d0a8dd8683fe73c808f329fe78b6ad8e1b1109b0b4"}
Sep 17 19:07:14.397043 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:14.396715    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:14.402744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:14.402720    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm"
Sep 17 19:07:14.416541 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:14.416499    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-nwjbm" podStartSLOduration=2.354157309 podStartE2EDuration="4.416484695s" totalImagesPullingTime="2.062327386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:14.415105774 +0000 UTC m=+53.542847165" watchObservedRunningTime="2026-09-17 19:07:14.416484695 +0000 UTC m=+53.544226096"
Sep 17 19:07:15.403878 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:15.403791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs" event={"ID":"9c6ceaf7-95b8-41f7-812e-8f93c1636316","Type":"ContainerStarted","Data":"9d979a9966b40d9fee92d7f5beb0de6d2c7f0532a3e2c43152c56241e3fea31f"}
Sep 17 19:07:15.404289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:15.403878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs" event={"ID":"9c6ceaf7-95b8-41f7-812e-8f93c1636316","Type":"ContainerStarted","Data":"eb32e363ab724901b15894ba1db58a2a794dc9742a03e1b978253de7a385708b"}
Sep 17 19:07:15.404289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:15.403926    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs" event={"ID":"9c6ceaf7-95b8-41f7-812e-8f93c1636316","Type":"ContainerStarted","Data":"70847475a82be1b7f05c0a38dfc71e03c38d59f3d9c3bffe3a51e39dc7984e66"}
Sep 17 19:07:15.404289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:15.403973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:15.408151 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:15.408124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"33731d5a-c3dc-4464-ba64-686c33118bd9","Type":"ContainerStarted","Data":"a9ea8ea73992b6a0adc20f8a3858aaa337b8ef0a67193bd69329d9978abd90ec"}
Sep 17 19:07:15.429705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:15.429661    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs" podStartSLOduration=2.837360177 podStartE2EDuration="8.429643991s" totalImagesPullingTime="5.592283814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:15.427584103 +0000 UTC m=+54.555325505" watchObservedRunningTime="2026-09-17 19:07:15.429643991 +0000 UTC m=+54.557385394"
Sep 17 19:07:15.460121 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:15.460069    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.357966448 podStartE2EDuration="9.460052796s" totalImagesPullingTime="5.084609368s" totalInitContainerRuntime="17.47698ms" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:15.45811837 +0000 UTC m=+54.585859770" watchObservedRunningTime="2026-09-17 19:07:15.460052796 +0000 UTC m=+54.587794235"
Sep 17 19:07:18.422545 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:18.422516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9ca9fe-6839-4aaf-808e-4ba059036268","Type":"ContainerStarted","Data":"9cd809fc66469fb7981d26e7c8360ab707c4f39bd237f6755ef455007c7803eb"}
Sep 17 19:07:18.422841 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:18.422582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9ca9fe-6839-4aaf-808e-4ba059036268","Type":"ContainerStarted","Data":"dc9bebb8e984328630773e713d5e3991f5e209b92082ff089a816576acfcf084"}
Sep 17 19:07:18.422841 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:18.422610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9ca9fe-6839-4aaf-808e-4ba059036268","Type":"ContainerStarted","Data":"e7f2486d5850ed381713d20f7cab0b9fbf9eb8c89df9eacd3303d3d3290759e4"}
Sep 17 19:07:18.422841 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:18.422663    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9ca9fe-6839-4aaf-808e-4ba059036268","Type":"ContainerStarted","Data":"6c2ecb4f7b9df84df2a92f7c6aa0069c2116a39b4f0483c2b74815aab6d139b5"}
Sep 17 19:07:19.430088 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:19.430036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9ca9fe-6839-4aaf-808e-4ba059036268","Type":"ContainerStarted","Data":"3e17eca2cdfdfbaeccb7872966b3af91af87db359a16be8bd461cf16d0c3d6a8"}
Sep 17 19:07:19.430479 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:19.430123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9e9ca9fe-6839-4aaf-808e-4ba059036268","Type":"ContainerStarted","Data":"a7cf51c5a24ea87bc4f00b6a145392773c45163572044d046c9894413db455b8"}
Sep 17 19:07:19.469665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:19.469592    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.006422169 podStartE2EDuration="7.469575846s" totalImagesPullingTime="4.425919101s" totalInitContainerRuntime="37.234576ms" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:19.468073419 +0000 UTC m=+58.595814860" watchObservedRunningTime="2026-09-17 19:07:19.469575846 +0000 UTC m=+58.597317258"
Sep 17 19:07:21.418312 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:21.418282    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5677f75b99-pz5zs"
Sep 17 19:07:22.273232 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:22.273200    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtf4n"
Sep 17 19:07:22.554736 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:22.554650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:28.839289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.839249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:07:28.839289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.839291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:07:28.842212 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.842183    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:07:28.842893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.842875    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:07:28.852117 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.852096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5da89ee3-23c3-4a18-a046-0dde553add07-metrics-certs\") pod \"network-metrics-daemon-g886c\" (UID: \"5da89ee3-23c3-4a18-a046-0dde553add07\") " pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:07:28.852289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.852270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e92de3-419a-4bff-858a-d15f9509c77a-original-pull-secret\") pod \"global-pull-secret-syncer-xhl95\" (UID: \"b5e92de3-419a-4bff-858a-d15f9509c77a\") " pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:07:28.940577 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.940548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:07:28.942954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.942938    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:07:28.953436 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.953409    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:07:28.954494 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.954479    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sjq9g\""
Sep 17 19:07:28.962383 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.962369    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g886c"
Sep 17 19:07:28.964541 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:28.964518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvd62\" (UniqueName: \"kubernetes.io/projected/abc81dc4-5b90-4a46-ad49-35d0fb866226-kube-api-access-vvd62\") pod \"network-check-target-vdcz8\" (UID: \"abc81dc4-5b90-4a46-ad49-35d0fb866226\") " pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:07:29.148037 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:29.147968    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jnl2p\""
Sep 17 19:07:29.150875 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:29.150862    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhl95"
Sep 17 19:07:29.156121 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:29.156097    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:07:29.234571 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:29.234545    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g886c"]
Sep 17 19:07:29.236988 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:29.236954    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da89ee3_23c3_4a18_a046_0dde553add07.slice/crio-0f2e416188b2afc2e43555c9c4378fe2fe871cd6aa2ff9743aacc8953f44b3bd WatchSource:0}: Error finding container 0f2e416188b2afc2e43555c9c4378fe2fe871cd6aa2ff9743aacc8953f44b3bd: Status 404 returned error can't find the container with id 0f2e416188b2afc2e43555c9c4378fe2fe871cd6aa2ff9743aacc8953f44b3bd
Sep 17 19:07:29.419192 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:29.419171    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhl95"]
Sep 17 19:07:29.421583 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:29.421547    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e92de3_419a_4bff_858a_d15f9509c77a.slice/crio-af90bd465ae03f2d29dc50ca42bbcfdf7e963f6f0b05d38311288dcea74fcd2d WatchSource:0}: Error finding container af90bd465ae03f2d29dc50ca42bbcfdf7e963f6f0b05d38311288dcea74fcd2d: Status 404 returned error can't find the container with id af90bd465ae03f2d29dc50ca42bbcfdf7e963f6f0b05d38311288dcea74fcd2d
Sep 17 19:07:29.430451 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:29.430400    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdcz8"]
Sep 17 19:07:29.432975 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:07:29.432953    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc81dc4_5b90_4a46_ad49_35d0fb866226.slice/crio-3d6d371fc512c75ab3aa8577ea8145063086ca53f19c99acfe8f0483cb59c4d5 WatchSource:0}: Error finding container 3d6d371fc512c75ab3aa8577ea8145063086ca53f19c99acfe8f0483cb59c4d5: Status 404 returned error can't find the container with id 3d6d371fc512c75ab3aa8577ea8145063086ca53f19c99acfe8f0483cb59c4d5
Sep 17 19:07:29.472200 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:29.472174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdcz8" event={"ID":"abc81dc4-5b90-4a46-ad49-35d0fb866226","Type":"ContainerStarted","Data":"3d6d371fc512c75ab3aa8577ea8145063086ca53f19c99acfe8f0483cb59c4d5"}
Sep 17 19:07:29.473367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:29.473339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g886c" event={"ID":"5da89ee3-23c3-4a18-a046-0dde553add07","Type":"ContainerStarted","Data":"0f2e416188b2afc2e43555c9c4378fe2fe871cd6aa2ff9743aacc8953f44b3bd"}
Sep 17 19:07:29.474344 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:29.474327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhl95" event={"ID":"b5e92de3-419a-4bff-858a-d15f9509c77a","Type":"ContainerStarted","Data":"af90bd465ae03f2d29dc50ca42bbcfdf7e963f6f0b05d38311288dcea74fcd2d"}
Sep 17 19:07:30.482031 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:30.481989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g886c" event={"ID":"5da89ee3-23c3-4a18-a046-0dde553add07","Type":"ContainerStarted","Data":"ffdcce6a3357c4b4a1e4130d89134e89ac8cb92afd83e52aa77fc85b69e12c06"}
Sep 17 19:07:30.482680 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:30.482112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g886c" event={"ID":"5da89ee3-23c3-4a18-a046-0dde553add07","Type":"ContainerStarted","Data":"aa8fa735314e3191a28ef1e733da883266ac974d67427aa9585fd9df63248ab7"}
Sep 17 19:07:30.519466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:30.519412    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g886c" podStartSLOduration=66.603928167 podStartE2EDuration="1m7.519391081s" totalImagesPullingTime="915.462914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:30.517220617 +0000 UTC m=+69.644962029" watchObservedRunningTime="2026-09-17 19:07:30.519391081 +0000 UTC m=+69.647132536"
Sep 17 19:07:34.502632 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:34.502581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhl95" event={"ID":"b5e92de3-419a-4bff-858a-d15f9509c77a","Type":"ContainerStarted","Data":"d98110ac20529afc3320fa4d020446ee30e91451228ee5ab5b7622eb49def0eb"}
Sep 17 19:07:34.504208 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:34.504186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdcz8" event={"ID":"abc81dc4-5b90-4a46-ad49-35d0fb866226","Type":"ContainerStarted","Data":"d1cc5e1892ba89efb96f48bdb427abc68e8139abf674f6cc7458a20b15c825a3"}
Sep 17 19:07:34.504344 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:34.504323    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:07:34.529478 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:34.529423    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xhl95" podStartSLOduration=66.996801457 podStartE2EDuration="1m11.529403867s" totalImagesPullingTime="4.53260241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:34.529070453 +0000 UTC m=+73.656811855" watchObservedRunningTime="2026-09-17 19:07:34.529403867 +0000 UTC m=+73.657145268"
Sep 17 19:07:34.561745 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:34.561685    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vdcz8" podStartSLOduration=67.044427689 podStartE2EDuration="1m11.561660851s" totalImagesPullingTime="4.517233162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:34.556699318 +0000 UTC m=+73.684440720" watchObservedRunningTime="2026-09-17 19:07:34.561660851 +0000 UTC m=+73.689402253"
Sep 17 19:07:50.891248 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:50.891210    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:50.911470 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:50.911445    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:07:51.588165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:07:51.588141    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 19:08:05.522018 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:08:05.517828    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vdcz8"
Sep 17 19:11:11.899606 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:11.899575    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2fmrq"]
Sep 17 19:11:11.905803 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:11.905774    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:11:11.908102 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:11.908077    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 19:11:11.908224 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:11.908083    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 19:11:11.908224 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:11.908201    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-c4btc\""
Sep 17 19:11:11.908862 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:11.908843    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 19:11:11.911065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:11.911039    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2fmrq"]
Sep 17 19:11:11.986918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:11.986888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnr4\" (UniqueName: \"kubernetes.io/projected/5c4882cf-2553-458d-94df-0ffdce8f0427-kube-api-access-qmnr4\") pod \"seaweedfs-64568bcd49-2fmrq\" (UID: \"5c4882cf-2553-458d-94df-0ffdce8f0427\") " pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:11:11.987077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:11.986932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5c4882cf-2553-458d-94df-0ffdce8f0427-data\") pod \"seaweedfs-64568bcd49-2fmrq\" (UID: \"5c4882cf-2553-458d-94df-0ffdce8f0427\") " pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:11:12.087845 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:12.087816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnr4\" (UniqueName: \"kubernetes.io/projected/5c4882cf-2553-458d-94df-0ffdce8f0427-kube-api-access-qmnr4\") pod \"seaweedfs-64568bcd49-2fmrq\" (UID: \"5c4882cf-2553-458d-94df-0ffdce8f0427\") " pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:11:12.087974 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:12.087861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5c4882cf-2553-458d-94df-0ffdce8f0427-data\") pod \"seaweedfs-64568bcd49-2fmrq\" (UID: \"5c4882cf-2553-458d-94df-0ffdce8f0427\") " pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:11:12.088801 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:12.088786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5c4882cf-2553-458d-94df-0ffdce8f0427-data\") pod \"seaweedfs-64568bcd49-2fmrq\" (UID: \"5c4882cf-2553-458d-94df-0ffdce8f0427\") " pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:11:12.112473 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:12.112449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnr4\" (UniqueName: \"kubernetes.io/projected/5c4882cf-2553-458d-94df-0ffdce8f0427-kube-api-access-qmnr4\") pod \"seaweedfs-64568bcd49-2fmrq\" (UID: \"5c4882cf-2553-458d-94df-0ffdce8f0427\") " pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:11:12.216605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:12.216578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:11:12.484017 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:12.483948    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2fmrq"]
Sep 17 19:11:12.486767 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:11:12.486742    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4882cf_2553_458d_94df_0ffdce8f0427.slice/crio-f4e71f8181a85c55977c8cc53281d8c620912d91a6d3a92e0b32ff63a5b9030b WatchSource:0}: Error finding container f4e71f8181a85c55977c8cc53281d8c620912d91a6d3a92e0b32ff63a5b9030b: Status 404 returned error can't find the container with id f4e71f8181a85c55977c8cc53281d8c620912d91a6d3a92e0b32ff63a5b9030b
Sep 17 19:11:13.336881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:13.336846    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2fmrq" event={"ID":"5c4882cf-2553-458d-94df-0ffdce8f0427","Type":"ContainerStarted","Data":"f4e71f8181a85c55977c8cc53281d8c620912d91a6d3a92e0b32ff63a5b9030b"}
Sep 17 19:11:17.354032 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:17.353997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2fmrq" event={"ID":"5c4882cf-2553-458d-94df-0ffdce8f0427","Type":"ContainerStarted","Data":"40cedc46e89b13bc0ca73c5e22f1d957fa98e0ed8bf1b18770f45e976c9c289a"}
Sep 17 19:11:17.354514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:17.354112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:11:17.368798 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:17.368756    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2fmrq" podStartSLOduration=2.313847507 podStartE2EDuration="6.368741717s" totalImagesPullingTime="4.05489421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:17.368331327 +0000 UTC m=+296.496072729" watchObservedRunningTime="2026-09-17 19:11:17.368741717 +0000 UTC m=+296.496483117"
Sep 17 19:11:22.897120 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:22.897090    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:11:22.900974 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:22.900955    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:11:22.901520 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:22.901498    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 19:11:23.360667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:11:23.360610    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2fmrq"
Sep 17 19:12:47.616545 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.616510    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-kgjbc"]
Sep 17 19:12:47.619864 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.619845    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kgjbc"
Sep 17 19:12:47.622366 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.622349    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 19:12:47.628807 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.628785    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kgjbc"]
Sep 17 19:12:47.737014 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.736980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ff399-31c0-4719-a368-341bb74b9153-model-cache\") pod \"s3-init-kgjbc\" (UID: \"674ff399-31c0-4719-a368-341bb74b9153\") " pod="kserve/s3-init-kgjbc"
Sep 17 19:12:47.737169 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.737025    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4mdj\" (UniqueName: \"kubernetes.io/projected/674ff399-31c0-4719-a368-341bb74b9153-kube-api-access-d4mdj\") pod \"s3-init-kgjbc\" (UID: \"674ff399-31c0-4719-a368-341bb74b9153\") " pod="kserve/s3-init-kgjbc"
Sep 17 19:12:47.837728 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.837692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ff399-31c0-4719-a368-341bb74b9153-model-cache\") pod \"s3-init-kgjbc\" (UID: \"674ff399-31c0-4719-a368-341bb74b9153\") " pod="kserve/s3-init-kgjbc"
Sep 17 19:12:47.837889 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.837742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mdj\" (UniqueName: \"kubernetes.io/projected/674ff399-31c0-4719-a368-341bb74b9153-kube-api-access-d4mdj\") pod \"s3-init-kgjbc\" (UID: \"674ff399-31c0-4719-a368-341bb74b9153\") " pod="kserve/s3-init-kgjbc"
Sep 17 19:12:47.838089 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.838070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ff399-31c0-4719-a368-341bb74b9153-model-cache\") pod \"s3-init-kgjbc\" (UID: \"674ff399-31c0-4719-a368-341bb74b9153\") " pod="kserve/s3-init-kgjbc"
Sep 17 19:12:47.848563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.848527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mdj\" (UniqueName: \"kubernetes.io/projected/674ff399-31c0-4719-a368-341bb74b9153-kube-api-access-d4mdj\") pod \"s3-init-kgjbc\" (UID: \"674ff399-31c0-4719-a368-341bb74b9153\") " pod="kserve/s3-init-kgjbc"
Sep 17 19:12:47.941224 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:47.941198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kgjbc"
Sep 17 19:12:48.211565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:48.211540    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kgjbc"]
Sep 17 19:12:48.214095 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:12:48.214062    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod674ff399_31c0_4719_a368_341bb74b9153.slice/crio-e3eb7e3c160281ff3d61222903fc4874ec1b486edae4337a8cb1f0fec9056a70 WatchSource:0}: Error finding container e3eb7e3c160281ff3d61222903fc4874ec1b486edae4337a8cb1f0fec9056a70: Status 404 returned error can't find the container with id e3eb7e3c160281ff3d61222903fc4874ec1b486edae4337a8cb1f0fec9056a70
Sep 17 19:12:48.215850 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:48.215832    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:12:48.688340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:48.688300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kgjbc" event={"ID":"674ff399-31c0-4719-a368-341bb74b9153","Type":"ContainerStarted","Data":"e3eb7e3c160281ff3d61222903fc4874ec1b486edae4337a8cb1f0fec9056a70"}
Sep 17 19:12:52.706959 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:52.706926    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kgjbc" event={"ID":"674ff399-31c0-4719-a368-341bb74b9153","Type":"ContainerStarted","Data":"588a14a1b50856a467aaeed4cac182039c71cedb4157022ecb025e9800f6e19a"}
Sep 17 19:12:54.716029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:54.716000    2583 generic.go:356] "Generic (PLEG): container finished" podID="674ff399-31c0-4719-a368-341bb74b9153" containerID="588a14a1b50856a467aaeed4cac182039c71cedb4157022ecb025e9800f6e19a" exitCode=0
Sep 17 19:12:54.716413 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:54.716079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kgjbc" event={"ID":"674ff399-31c0-4719-a368-341bb74b9153","Type":"ContainerDied","Data":"588a14a1b50856a467aaeed4cac182039c71cedb4157022ecb025e9800f6e19a"}
Sep 17 19:12:59.742476 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:59.742441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kgjbc" event={"ID":"674ff399-31c0-4719-a368-341bb74b9153","Type":"ContainerStarted","Data":"0b4d686da710e677a33f3cc1ecb00450ed16234fab84589295ae2d48b60f7edc"}
Sep 17 19:12:59.761040 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:12:59.760985    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-kgjbc" podStartSLOduration=2.762321651 podStartE2EDuration="12.760968787s" totalImagesPullingTime="8.222774931s" totalInitContainerRuntime="1.775872205s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:59.76053575 +0000 UTC m=+398.888277150" watchObservedRunningTime="2026-09-17 19:12:59.760968787 +0000 UTC m=+398.888710187"
Sep 17 19:14:03.995537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:03.995507    2583 generic.go:356] "Generic (PLEG): container finished" podID="674ff399-31c0-4719-a368-341bb74b9153" containerID="0b4d686da710e677a33f3cc1ecb00450ed16234fab84589295ae2d48b60f7edc" exitCode=0
Sep 17 19:14:03.995952 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:03.995592    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kgjbc" event={"ID":"674ff399-31c0-4719-a368-341bb74b9153","Type":"ContainerDied","Data":"0b4d686da710e677a33f3cc1ecb00450ed16234fab84589295ae2d48b60f7edc"}
Sep 17 19:14:05.130719 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:05.130694    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kgjbc"
Sep 17 19:14:05.259478 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:05.259393    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4mdj\" (UniqueName: \"kubernetes.io/projected/674ff399-31c0-4719-a368-341bb74b9153-kube-api-access-d4mdj\") pod \"674ff399-31c0-4719-a368-341bb74b9153\" (UID: \"674ff399-31c0-4719-a368-341bb74b9153\") "
Sep 17 19:14:05.259478 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:05.259446    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ff399-31c0-4719-a368-341bb74b9153-model-cache\") pod \"674ff399-31c0-4719-a368-341bb74b9153\" (UID: \"674ff399-31c0-4719-a368-341bb74b9153\") "
Sep 17 19:14:05.261745 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:05.261704    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674ff399-31c0-4719-a368-341bb74b9153-kube-api-access-d4mdj" (OuterVolumeSpecName: "kube-api-access-d4mdj") pod "674ff399-31c0-4719-a368-341bb74b9153" (UID: "674ff399-31c0-4719-a368-341bb74b9153"). InnerVolumeSpecName "kube-api-access-d4mdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:05.278196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:05.278171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674ff399-31c0-4719-a368-341bb74b9153-model-cache" (OuterVolumeSpecName: "model-cache") pod "674ff399-31c0-4719-a368-341bb74b9153" (UID: "674ff399-31c0-4719-a368-341bb74b9153"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:05.360175 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:05.360144    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4mdj\" (UniqueName: \"kubernetes.io/projected/674ff399-31c0-4719-a368-341bb74b9153-kube-api-access-d4mdj\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:14:05.360175 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:05.360170    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ff399-31c0-4719-a368-341bb74b9153-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:14:06.004312 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:06.004285    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kgjbc"
Sep 17 19:14:06.004486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:06.004278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kgjbc" event={"ID":"674ff399-31c0-4719-a368-341bb74b9153","Type":"ContainerDied","Data":"e3eb7e3c160281ff3d61222903fc4874ec1b486edae4337a8cb1f0fec9056a70"}
Sep 17 19:14:06.004486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:06.004390    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3eb7e3c160281ff3d61222903fc4874ec1b486edae4337a8cb1f0fec9056a70"
Sep 17 19:14:42.616174 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.616146    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-2lp4j"]
Sep 17 19:14:42.616651 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.616454    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674ff399-31c0-4719-a368-341bb74b9153" containerName="download-hf-model"
Sep 17 19:14:42.616651 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.616465    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="674ff399-31c0-4719-a368-341bb74b9153" containerName="download-hf-model"
Sep 17 19:14:42.616651 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.616482    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674ff399-31c0-4719-a368-341bb74b9153" containerName="s3-init"
Sep 17 19:14:42.616651 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.616487    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="674ff399-31c0-4719-a368-341bb74b9153" containerName="s3-init"
Sep 17 19:14:42.616651 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.616543    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="674ff399-31c0-4719-a368-341bb74b9153" containerName="s3-init"
Sep 17 19:14:42.619331 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.619314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:42.621540 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.621515    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 19:14:42.621607 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.621516    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 19:14:42.627012 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.626990    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-2lp4j"]
Sep 17 19:14:42.736676 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.736649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6s8\" (UniqueName: \"kubernetes.io/projected/fd741661-cca6-484b-8133-10cebfd86a19-kube-api-access-7v6s8\") pod \"s3-tls-init-custom-2lp4j\" (UID: \"fd741661-cca6-484b-8133-10cebfd86a19\") " pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:42.736676 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.736678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd741661-cca6-484b-8133-10cebfd86a19-model-cache\") pod \"s3-tls-init-custom-2lp4j\" (UID: \"fd741661-cca6-484b-8133-10cebfd86a19\") " pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:42.837391 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.837360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6s8\" (UniqueName: \"kubernetes.io/projected/fd741661-cca6-484b-8133-10cebfd86a19-kube-api-access-7v6s8\") pod \"s3-tls-init-custom-2lp4j\" (UID: \"fd741661-cca6-484b-8133-10cebfd86a19\") " pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:42.837391 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.837391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd741661-cca6-484b-8133-10cebfd86a19-model-cache\") pod \"s3-tls-init-custom-2lp4j\" (UID: \"fd741661-cca6-484b-8133-10cebfd86a19\") " pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:42.837745 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.837728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd741661-cca6-484b-8133-10cebfd86a19-model-cache\") pod \"s3-tls-init-custom-2lp4j\" (UID: \"fd741661-cca6-484b-8133-10cebfd86a19\") " pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:42.845566 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.845542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6s8\" (UniqueName: \"kubernetes.io/projected/fd741661-cca6-484b-8133-10cebfd86a19-kube-api-access-7v6s8\") pod \"s3-tls-init-custom-2lp4j\" (UID: \"fd741661-cca6-484b-8133-10cebfd86a19\") " pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:42.938946 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:42.938921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:43.212669 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:43.210158    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-2lp4j"]
Sep 17 19:14:43.216359 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:14:43.216328    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd741661_cca6_484b_8133_10cebfd86a19.slice/crio-bddd0b35c16fe62a386c7f56159dc57d68be83a021ed54f1095396da3bc0e656 WatchSource:0}: Error finding container bddd0b35c16fe62a386c7f56159dc57d68be83a021ed54f1095396da3bc0e656: Status 404 returned error can't find the container with id bddd0b35c16fe62a386c7f56159dc57d68be83a021ed54f1095396da3bc0e656
Sep 17 19:14:44.150797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:44.150753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2lp4j" event={"ID":"fd741661-cca6-484b-8133-10cebfd86a19","Type":"ContainerStarted","Data":"56452498efb2c6755f766cb30b529d5794a7604d52391a8e11d89b1402e2a0c0"}
Sep 17 19:14:44.151163 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:44.150803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2lp4j" event={"ID":"fd741661-cca6-484b-8133-10cebfd86a19","Type":"ContainerStarted","Data":"bddd0b35c16fe62a386c7f56159dc57d68be83a021ed54f1095396da3bc0e656"}
Sep 17 19:14:46.159871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:46.159838    2583 generic.go:356] "Generic (PLEG): container finished" podID="fd741661-cca6-484b-8133-10cebfd86a19" containerID="56452498efb2c6755f766cb30b529d5794a7604d52391a8e11d89b1402e2a0c0" exitCode=0
Sep 17 19:14:46.160224 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:46.159899    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2lp4j" event={"ID":"fd741661-cca6-484b-8133-10cebfd86a19","Type":"ContainerDied","Data":"56452498efb2c6755f766cb30b529d5794a7604d52391a8e11d89b1402e2a0c0"}
Sep 17 19:14:47.164978 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:47.164945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2lp4j" event={"ID":"fd741661-cca6-484b-8133-10cebfd86a19","Type":"ContainerStarted","Data":"335897bf723c7d1b22c480ce6fdacaa4542bc0a30c30d9c2aee8298aec2e35b6"}
Sep 17 19:14:47.182085 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:47.182034    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-2lp4j" podStartSLOduration=3.006678204 podStartE2EDuration="5.182014841s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.175336637s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:47.181672456 +0000 UTC m=+506.309413861" watchObservedRunningTime="2026-09-17 19:14:47.182014841 +0000 UTC m=+506.309756239"
Sep 17 19:14:53.193879 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:53.193845    2583 generic.go:356] "Generic (PLEG): container finished" podID="fd741661-cca6-484b-8133-10cebfd86a19" containerID="335897bf723c7d1b22c480ce6fdacaa4542bc0a30c30d9c2aee8298aec2e35b6" exitCode=0
Sep 17 19:14:53.194275 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:53.193887    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2lp4j" event={"ID":"fd741661-cca6-484b-8133-10cebfd86a19","Type":"ContainerDied","Data":"335897bf723c7d1b22c480ce6fdacaa4542bc0a30c30d9c2aee8298aec2e35b6"}
Sep 17 19:14:54.326973 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:54.326943    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:54.437249 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:54.437214    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v6s8\" (UniqueName: \"kubernetes.io/projected/fd741661-cca6-484b-8133-10cebfd86a19-kube-api-access-7v6s8\") pod \"fd741661-cca6-484b-8133-10cebfd86a19\" (UID: \"fd741661-cca6-484b-8133-10cebfd86a19\") "
Sep 17 19:14:54.437429 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:54.437275    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd741661-cca6-484b-8133-10cebfd86a19-model-cache\") pod \"fd741661-cca6-484b-8133-10cebfd86a19\" (UID: \"fd741661-cca6-484b-8133-10cebfd86a19\") "
Sep 17 19:14:54.439381 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:54.439345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd741661-cca6-484b-8133-10cebfd86a19-kube-api-access-7v6s8" (OuterVolumeSpecName: "kube-api-access-7v6s8") pod "fd741661-cca6-484b-8133-10cebfd86a19" (UID: "fd741661-cca6-484b-8133-10cebfd86a19"). InnerVolumeSpecName "kube-api-access-7v6s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:54.459202 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:54.459154    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd741661-cca6-484b-8133-10cebfd86a19-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd741661-cca6-484b-8133-10cebfd86a19" (UID: "fd741661-cca6-484b-8133-10cebfd86a19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:54.537786 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:54.537767    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v6s8\" (UniqueName: \"kubernetes.io/projected/fd741661-cca6-484b-8133-10cebfd86a19-kube-api-access-7v6s8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:14:54.537786 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:54.537787    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd741661-cca6-484b-8133-10cebfd86a19-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:14:55.203248 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:55.203215    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2lp4j"
Sep 17 19:14:55.203248 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:55.203238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2lp4j" event={"ID":"fd741661-cca6-484b-8133-10cebfd86a19","Type":"ContainerDied","Data":"bddd0b35c16fe62a386c7f56159dc57d68be83a021ed54f1095396da3bc0e656"}
Sep 17 19:14:55.203479 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:55.203278    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bddd0b35c16fe62a386c7f56159dc57d68be83a021ed54f1095396da3bc0e656"
Sep 17 19:14:57.906728 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.906694    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-df4l7"]
Sep 17 19:14:57.907135 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.906995    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd741661-cca6-484b-8133-10cebfd86a19" containerName="s3-tls-init-custom"
Sep 17 19:14:57.907135 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.907006    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd741661-cca6-484b-8133-10cebfd86a19" containerName="s3-tls-init-custom"
Sep 17 19:14:57.907135 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.907031    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd741661-cca6-484b-8133-10cebfd86a19" containerName="download-hf-model"
Sep 17 19:14:57.907135 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.907037    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd741661-cca6-484b-8133-10cebfd86a19" containerName="download-hf-model"
Sep 17 19:14:57.907135 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.907082    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd741661-cca6-484b-8133-10cebfd86a19" containerName="s3-tls-init-custom"
Sep 17 19:14:57.910289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.910266    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:14:57.912480 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.912463    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 19:14:57.912480 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.912471    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 19:14:57.918594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:57.918573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-df4l7"]
Sep 17 19:14:58.068190 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:58.068159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b8b2c8-3011-43e2-8b30-033db4c476a4-model-cache\") pod \"s3-tls-init-serving-df4l7\" (UID: \"31b8b2c8-3011-43e2-8b30-033db4c476a4\") " pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:14:58.068363 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:58.068218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws479\" (UniqueName: \"kubernetes.io/projected/31b8b2c8-3011-43e2-8b30-033db4c476a4-kube-api-access-ws479\") pod \"s3-tls-init-serving-df4l7\" (UID: \"31b8b2c8-3011-43e2-8b30-033db4c476a4\") " pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:14:58.169288 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:58.169258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b8b2c8-3011-43e2-8b30-033db4c476a4-model-cache\") pod \"s3-tls-init-serving-df4l7\" (UID: \"31b8b2c8-3011-43e2-8b30-033db4c476a4\") " pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:14:58.169442 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:58.169319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws479\" (UniqueName: \"kubernetes.io/projected/31b8b2c8-3011-43e2-8b30-033db4c476a4-kube-api-access-ws479\") pod \"s3-tls-init-serving-df4l7\" (UID: \"31b8b2c8-3011-43e2-8b30-033db4c476a4\") " pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:14:58.169804 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:58.169785    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b8b2c8-3011-43e2-8b30-033db4c476a4-model-cache\") pod \"s3-tls-init-serving-df4l7\" (UID: \"31b8b2c8-3011-43e2-8b30-033db4c476a4\") " pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:14:58.177749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:58.177728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws479\" (UniqueName: \"kubernetes.io/projected/31b8b2c8-3011-43e2-8b30-033db4c476a4-kube-api-access-ws479\") pod \"s3-tls-init-serving-df4l7\" (UID: \"31b8b2c8-3011-43e2-8b30-033db4c476a4\") " pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:14:58.227474 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:58.227446    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:14:58.496875 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:58.496851    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-df4l7"]
Sep 17 19:14:58.499324 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:14:58.499284    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31b8b2c8_3011_43e2_8b30_033db4c476a4.slice/crio-6b420912ff2c7e240b72ba6ef861b52f4fe5360a26fb470b99974512de24a06a WatchSource:0}: Error finding container 6b420912ff2c7e240b72ba6ef861b52f4fe5360a26fb470b99974512de24a06a: Status 404 returned error can't find the container with id 6b420912ff2c7e240b72ba6ef861b52f4fe5360a26fb470b99974512de24a06a
Sep 17 19:14:59.219438 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:59.219402    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-df4l7" event={"ID":"31b8b2c8-3011-43e2-8b30-033db4c476a4","Type":"ContainerStarted","Data":"ec40852ac4c1c691ad83f569c563615e227c0def7de305269c07facf9359afc6"}
Sep 17 19:14:59.219438 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:14:59.219452    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-df4l7" event={"ID":"31b8b2c8-3011-43e2-8b30-033db4c476a4","Type":"ContainerStarted","Data":"6b420912ff2c7e240b72ba6ef861b52f4fe5360a26fb470b99974512de24a06a"}
Sep 17 19:15:01.229062 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:01.229028    2583 generic.go:356] "Generic (PLEG): container finished" podID="31b8b2c8-3011-43e2-8b30-033db4c476a4" containerID="ec40852ac4c1c691ad83f569c563615e227c0def7de305269c07facf9359afc6" exitCode=0
Sep 17 19:15:01.229456 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:01.229081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-df4l7" event={"ID":"31b8b2c8-3011-43e2-8b30-033db4c476a4","Type":"ContainerDied","Data":"ec40852ac4c1c691ad83f569c563615e227c0def7de305269c07facf9359afc6"}
Sep 17 19:15:02.235305 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:02.235234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-df4l7" event={"ID":"31b8b2c8-3011-43e2-8b30-033db4c476a4","Type":"ContainerStarted","Data":"8432161de4f8ced786439fe983f2f7c59be45253aac17002b745b1f7d4250387"}
Sep 17 19:15:02.253101 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:02.253054    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-df4l7" podStartSLOduration=3.083738909 podStartE2EDuration="5.253033304s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.169294395s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:02.251919118 +0000 UTC m=+521.379660519" watchObservedRunningTime="2026-09-17 19:15:02.253033304 +0000 UTC m=+521.380774707"
Sep 17 19:15:07.257735 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:07.257703    2583 generic.go:356] "Generic (PLEG): container finished" podID="31b8b2c8-3011-43e2-8b30-033db4c476a4" containerID="8432161de4f8ced786439fe983f2f7c59be45253aac17002b745b1f7d4250387" exitCode=0
Sep 17 19:15:07.258131 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:07.257778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-df4l7" event={"ID":"31b8b2c8-3011-43e2-8b30-033db4c476a4","Type":"ContainerDied","Data":"8432161de4f8ced786439fe983f2f7c59be45253aac17002b745b1f7d4250387"}
Sep 17 19:15:08.395509 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:08.395486    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:15:08.561263 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:08.561186    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws479\" (UniqueName: \"kubernetes.io/projected/31b8b2c8-3011-43e2-8b30-033db4c476a4-kube-api-access-ws479\") pod \"31b8b2c8-3011-43e2-8b30-033db4c476a4\" (UID: \"31b8b2c8-3011-43e2-8b30-033db4c476a4\") "
Sep 17 19:15:08.561263 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:08.561221    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b8b2c8-3011-43e2-8b30-033db4c476a4-model-cache\") pod \"31b8b2c8-3011-43e2-8b30-033db4c476a4\" (UID: \"31b8b2c8-3011-43e2-8b30-033db4c476a4\") "
Sep 17 19:15:08.563376 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:08.563345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b8b2c8-3011-43e2-8b30-033db4c476a4-kube-api-access-ws479" (OuterVolumeSpecName: "kube-api-access-ws479") pod "31b8b2c8-3011-43e2-8b30-033db4c476a4" (UID: "31b8b2c8-3011-43e2-8b30-033db4c476a4"). InnerVolumeSpecName "kube-api-access-ws479". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:15:08.583099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:08.583073    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b8b2c8-3011-43e2-8b30-033db4c476a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "31b8b2c8-3011-43e2-8b30-033db4c476a4" (UID: "31b8b2c8-3011-43e2-8b30-033db4c476a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:15:08.661828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:08.661801    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws479\" (UniqueName: \"kubernetes.io/projected/31b8b2c8-3011-43e2-8b30-033db4c476a4-kube-api-access-ws479\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:15:08.661828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:08.661825    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31b8b2c8-3011-43e2-8b30-033db4c476a4-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:15:09.269631 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:09.269585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-df4l7" event={"ID":"31b8b2c8-3011-43e2-8b30-033db4c476a4","Type":"ContainerDied","Data":"6b420912ff2c7e240b72ba6ef861b52f4fe5360a26fb470b99974512de24a06a"}
Sep 17 19:15:09.269791 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:09.269639    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b420912ff2c7e240b72ba6ef861b52f4fe5360a26fb470b99974512de24a06a"
Sep 17 19:15:09.269791 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:09.269639    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-df4l7"
Sep 17 19:15:22.022111 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.022071    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"]
Sep 17 19:15:22.022498 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.022381    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b8b2c8-3011-43e2-8b30-033db4c476a4" containerName="s3-tls-init-serving"
Sep 17 19:15:22.022498 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.022391    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b8b2c8-3011-43e2-8b30-033db4c476a4" containerName="s3-tls-init-serving"
Sep 17 19:15:22.022498 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.022406    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b8b2c8-3011-43e2-8b30-033db4c476a4" containerName="download-hf-model"
Sep 17 19:15:22.022498 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.022411    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b8b2c8-3011-43e2-8b30-033db4c476a4" containerName="download-hf-model"
Sep 17 19:15:22.022498 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.022461    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31b8b2c8-3011-43e2-8b30-033db4c476a4" containerName="s3-tls-init-serving"
Sep 17 19:15:22.025559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.025541    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.029370 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.029347    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 19:15:22.030042 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.030012    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 19:15:22.030158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.030071    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:15:22.030158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.030095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:15:22.030299 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.030285    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:15:22.030378 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.030324    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:15:22.030432 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.030327    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:15:22.043058 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.043034    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"]
Sep 17 19:15:22.065285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.065256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.065435 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.065293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcsgq\" (UniqueName: \"kubernetes.io/projected/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-api-access-zcsgq\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.065435 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.065388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.065545 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.065498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.065659 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.065642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.129377 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.129337    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"]
Sep 17 19:15:22.132849 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.132828    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.135187 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.135158    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-2b7285\""
Sep 17 19:15:22.135307 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.135281    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2b7285-predictor-serving-cert\""
Sep 17 19:15:22.135416 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.135399    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2b7285-kube-rbac-proxy-sar-config\""
Sep 17 19:15:22.135491 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.135402    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-2b7285-dockercfg-7dwgn\""
Sep 17 19:15:22.142937 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.142916    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"]
Sep 17 19:15:22.166745 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166712    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaa4a1db-762b-4163-8be8-3d788f0fd805-proxy-tls\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.166871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6d54\" (UniqueName: \"kubernetes.io/projected/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-api-access-w6d54\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.166871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.166871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.166871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.167010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaa4a1db-762b-4163-8be8-3d788f0fd805-kserve-provision-location\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.167010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-cabundle-cert\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.167010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.167010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.167010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.166951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcsgq\" (UniqueName: \"kubernetes.io/projected/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-api-access-zcsgq\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.167211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.167163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.167640 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.167542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.167640 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.167564    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.169361 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.169341    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.174968 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.174948    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcsgq\" (UniqueName: \"kubernetes.io/projected/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-api-access-zcsgq\") pod \"isvc-sklearn-batcher-predictor-77774898d8-6bmh8\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.267720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.267686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6d54\" (UniqueName: \"kubernetes.io/projected/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-api-access-w6d54\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.267860 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.267751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.267860 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.267781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaa4a1db-762b-4163-8be8-3d788f0fd805-kserve-provision-location\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.267860 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.267807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-cabundle-cert\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.267860 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.267854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaa4a1db-762b-4163-8be8-3d788f0fd805-proxy-tls\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.268062 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:15:22.268006    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-serving-cert: secret "isvc-init-fail-2b7285-predictor-serving-cert" not found
Sep 17 19:15:22.268108 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:15:22.268091    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa4a1db-762b-4163-8be8-3d788f0fd805-proxy-tls podName:aaa4a1db-762b-4163-8be8-3d788f0fd805 nodeName:}" failed. No retries permitted until 2026-09-17 19:15:22.76807118 +0000 UTC m=+541.895812575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aaa4a1db-762b-4163-8be8-3d788f0fd805-proxy-tls") pod "isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7" (UID: "aaa4a1db-762b-4163-8be8-3d788f0fd805") : secret "isvc-init-fail-2b7285-predictor-serving-cert" not found
Sep 17 19:15:22.268186 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.268161    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaa4a1db-762b-4163-8be8-3d788f0fd805-kserve-provision-location\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.268483 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.268466    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-cabundle-cert\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.268534 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.268470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.278759 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.278708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6d54\" (UniqueName: \"kubernetes.io/projected/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-api-access-w6d54\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.336102 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.336063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:22.608811 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.608787    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"]
Sep 17 19:15:22.611099 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:15:22.611072    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12da2abd_f0e8_434c_a44e_fe52ddc4c28d.slice/crio-0bbed9bd6629f5788a9cc9f3d87bcfeab31549effda1ed5a6d9725eaadca2583 WatchSource:0}: Error finding container 0bbed9bd6629f5788a9cc9f3d87bcfeab31549effda1ed5a6d9725eaadca2583: Status 404 returned error can't find the container with id 0bbed9bd6629f5788a9cc9f3d87bcfeab31549effda1ed5a6d9725eaadca2583
Sep 17 19:15:22.772191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.772159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaa4a1db-762b-4163-8be8-3d788f0fd805-proxy-tls\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:22.774826 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:22.774801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaa4a1db-762b-4163-8be8-3d788f0fd805-proxy-tls\") pod \"isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") " pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:23.045690 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:23.045664    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-2b7285-dockercfg-7dwgn\""
Sep 17 19:15:23.053877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:23.053855    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:23.325074 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:23.324983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerStarted","Data":"61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172"}
Sep 17 19:15:23.325074 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:23.325052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerStarted","Data":"0bbed9bd6629f5788a9cc9f3d87bcfeab31549effda1ed5a6d9725eaadca2583"}
Sep 17 19:15:23.331691 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:23.331671    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"]
Sep 17 19:15:23.334220 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:15:23.334197    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa4a1db_762b_4163_8be8_3d788f0fd805.slice/crio-9eee6eabfa85710e41313fb50951d690115f3375dfb6d890eb51fb1115f91de9 WatchSource:0}: Error finding container 9eee6eabfa85710e41313fb50951d690115f3375dfb6d890eb51fb1115f91de9: Status 404 returned error can't find the container with id 9eee6eabfa85710e41313fb50951d690115f3375dfb6d890eb51fb1115f91de9
Sep 17 19:15:23.338742 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:23.338722    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-2b7285\""
Sep 17 19:15:24.330989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:24.330949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7" event={"ID":"aaa4a1db-762b-4163-8be8-3d788f0fd805","Type":"ContainerStarted","Data":"6e0858b90f8bd1951d05ed7e4848a4e22191ca794fa937577163fbdb203f9b12"}
Sep 17 19:15:24.331356 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:24.331044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7" event={"ID":"aaa4a1db-762b-4163-8be8-3d788f0fd805","Type":"ContainerStarted","Data":"9eee6eabfa85710e41313fb50951d690115f3375dfb6d890eb51fb1115f91de9"}
Sep 17 19:15:27.344679 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:27.344639    2583 generic.go:356] "Generic (PLEG): container finished" podID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerID="61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172" exitCode=0
Sep 17 19:15:27.345058 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:27.344710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerDied","Data":"61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172"}
Sep 17 19:15:29.363424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:29.363328    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7_aaa4a1db-762b-4163-8be8-3d788f0fd805/storage-initializer/0.log"
Sep 17 19:15:29.363424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:29.363373    2583 generic.go:356] "Generic (PLEG): container finished" podID="aaa4a1db-762b-4163-8be8-3d788f0fd805" containerID="6e0858b90f8bd1951d05ed7e4848a4e22191ca794fa937577163fbdb203f9b12" exitCode=1
Sep 17 19:15:29.363951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:29.363458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7" event={"ID":"aaa4a1db-762b-4163-8be8-3d788f0fd805","Type":"ContainerDied","Data":"6e0858b90f8bd1951d05ed7e4848a4e22191ca794fa937577163fbdb203f9b12"}
Sep 17 19:15:30.375058 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:30.374142    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7_aaa4a1db-762b-4163-8be8-3d788f0fd805/storage-initializer/0.log"
Sep 17 19:15:30.375058 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:30.374265    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7" event={"ID":"aaa4a1db-762b-4163-8be8-3d788f0fd805","Type":"ContainerStarted","Data":"0839cab7f31916e4768d8909feda47f29ee55998611d9a12e77ee7c7b60239b5"}
Sep 17 19:15:31.823145 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:31.822379    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"]
Sep 17 19:15:31.823145 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:31.822592    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7" podUID="aaa4a1db-762b-4163-8be8-3d788f0fd805" containerName="storage-initializer" containerID="cri-o://0839cab7f31916e4768d8909feda47f29ee55998611d9a12e77ee7c7b60239b5" gracePeriod=30
Sep 17 19:15:36.409516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:36.409433    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7_aaa4a1db-762b-4163-8be8-3d788f0fd805/storage-initializer/1.log"
Sep 17 19:15:36.409954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:36.409933    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7_aaa4a1db-762b-4163-8be8-3d788f0fd805/storage-initializer/0.log"
Sep 17 19:15:36.410018 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:36.409982    2583 generic.go:356] "Generic (PLEG): container finished" podID="aaa4a1db-762b-4163-8be8-3d788f0fd805" containerID="0839cab7f31916e4768d8909feda47f29ee55998611d9a12e77ee7c7b60239b5" exitCode=1
Sep 17 19:15:36.410082 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:36.410062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7" event={"ID":"aaa4a1db-762b-4163-8be8-3d788f0fd805","Type":"ContainerDied","Data":"0839cab7f31916e4768d8909feda47f29ee55998611d9a12e77ee7c7b60239b5"}
Sep 17 19:15:36.410145 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:36.410131    2583 scope.go:117] "RemoveContainer" containerID="6e0858b90f8bd1951d05ed7e4848a4e22191ca794fa937577163fbdb203f9b12"
Sep 17 19:15:37.019944 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.019909    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"]
Sep 17 19:15:37.058442 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.058413    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"]
Sep 17 19:15:37.058638 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.058537    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.061713 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.061689    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 19:15:37.061857 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.061688    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 19:15:37.206443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.206407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.206443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.206446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.206730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.206497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.206730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.206555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8dqj\" (UniqueName: \"kubernetes.io/projected/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-api-access-q8dqj\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.206730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.206686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.307365 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.307272    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.307365 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.307334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.307601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.307389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.307601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.307417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dqj\" (UniqueName: \"kubernetes.io/projected/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-api-access-q8dqj\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.307601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.307502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.307878 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.307857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.308301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.308270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.308436 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.308398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.310147 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.310125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.316180 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.316157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dqj\" (UniqueName: \"kubernetes.io/projected/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-api-access-q8dqj\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-9mf4b\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:37.371500 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:37.371463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:15:40.580797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.580769    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7_aaa4a1db-762b-4163-8be8-3d788f0fd805/storage-initializer/1.log"
Sep 17 19:15:40.583680 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.583658    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:40.740529 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.740500    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaa4a1db-762b-4163-8be8-3d788f0fd805-kserve-provision-location\") pod \"aaa4a1db-762b-4163-8be8-3d788f0fd805\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") "
Sep 17 19:15:40.740648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.740537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-cabundle-cert\") pod \"aaa4a1db-762b-4163-8be8-3d788f0fd805\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") "
Sep 17 19:15:40.740648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.740569    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaa4a1db-762b-4163-8be8-3d788f0fd805-proxy-tls\") pod \"aaa4a1db-762b-4163-8be8-3d788f0fd805\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") "
Sep 17 19:15:40.740648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.740584    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6d54\" (UniqueName: \"kubernetes.io/projected/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-api-access-w6d54\") pod \"aaa4a1db-762b-4163-8be8-3d788f0fd805\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") "
Sep 17 19:15:40.740840 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.740744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-rbac-proxy-sar-config\") pod \"aaa4a1db-762b-4163-8be8-3d788f0fd805\" (UID: \"aaa4a1db-762b-4163-8be8-3d788f0fd805\") "
Sep 17 19:15:40.740959 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.740742    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa4a1db-762b-4163-8be8-3d788f0fd805-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aaa4a1db-762b-4163-8be8-3d788f0fd805" (UID: "aaa4a1db-762b-4163-8be8-3d788f0fd805"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:15:40.741017 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.740986    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aaa4a1db-762b-4163-8be8-3d788f0fd805" (UID: "aaa4a1db-762b-4163-8be8-3d788f0fd805"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:15:40.741094 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.741070    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aaa4a1db-762b-4163-8be8-3d788f0fd805" (UID: "aaa4a1db-762b-4163-8be8-3d788f0fd805"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:15:40.741356 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.741092    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaa4a1db-762b-4163-8be8-3d788f0fd805-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:15:40.741356 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.741110    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:15:40.742780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.742756    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa4a1db-762b-4163-8be8-3d788f0fd805-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aaa4a1db-762b-4163-8be8-3d788f0fd805" (UID: "aaa4a1db-762b-4163-8be8-3d788f0fd805"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:15:40.743174 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.743154    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-api-access-w6d54" (OuterVolumeSpecName: "kube-api-access-w6d54") pod "aaa4a1db-762b-4163-8be8-3d788f0fd805" (UID: "aaa4a1db-762b-4163-8be8-3d788f0fd805"). InnerVolumeSpecName "kube-api-access-w6d54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:15:40.836165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.836141    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"]
Sep 17 19:15:40.838679 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:15:40.838654    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca96e10_efbc_4eea_b6e4_7ffa8a7c9515.slice/crio-d43b2e3d675ba23a0b08f5af4fa167f483ed9a540157d979a6b90e8ea4786b1e WatchSource:0}: Error finding container d43b2e3d675ba23a0b08f5af4fa167f483ed9a540157d979a6b90e8ea4786b1e: Status 404 returned error can't find the container with id d43b2e3d675ba23a0b08f5af4fa167f483ed9a540157d979a6b90e8ea4786b1e
Sep 17 19:15:40.841534 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.841517    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:15:40.841596 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.841537    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaa4a1db-762b-4163-8be8-3d788f0fd805-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:15:40.841596 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:40.841546    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6d54\" (UniqueName: \"kubernetes.io/projected/aaa4a1db-762b-4163-8be8-3d788f0fd805-kube-api-access-w6d54\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:15:41.438738 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:41.438696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerStarted","Data":"36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680"}
Sep 17 19:15:41.440579 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:41.440551    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7_aaa4a1db-762b-4163-8be8-3d788f0fd805/storage-initializer/1.log"
Sep 17 19:15:41.442102 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:41.440733    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"
Sep 17 19:15:41.442102 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:41.441301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7" event={"ID":"aaa4a1db-762b-4163-8be8-3d788f0fd805","Type":"ContainerDied","Data":"9eee6eabfa85710e41313fb50951d690115f3375dfb6d890eb51fb1115f91de9"}
Sep 17 19:15:41.442102 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:41.441359    2583 scope.go:117] "RemoveContainer" containerID="0839cab7f31916e4768d8909feda47f29ee55998611d9a12e77ee7c7b60239b5"
Sep 17 19:15:41.445283 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:41.445241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" event={"ID":"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515","Type":"ContainerStarted","Data":"3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0"}
Sep 17 19:15:41.445385 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:41.445303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" event={"ID":"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515","Type":"ContainerStarted","Data":"d43b2e3d675ba23a0b08f5af4fa167f483ed9a540157d979a6b90e8ea4786b1e"}
Sep 17 19:15:41.496563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:41.496526    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"]
Sep 17 19:15:41.505443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:41.505382    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2b7285-predictor-69c48b8788-tb6l7"]
Sep 17 19:15:43.040756 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:43.040730    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaa4a1db-762b-4163-8be8-3d788f0fd805" path="/var/lib/kubelet/pods/aaa4a1db-762b-4163-8be8-3d788f0fd805/volumes"
Sep 17 19:15:43.457191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:43.457158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerStarted","Data":"cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7"}
Sep 17 19:15:45.469280 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:45.469248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerStarted","Data":"074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f"}
Sep 17 19:15:45.469737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:45.469463    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:45.469737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:45.469519    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:45.469737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:45.469552    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:45.470865 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:45.470845    2583 generic.go:356] "Generic (PLEG): container finished" podID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerID="3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0" exitCode=0
Sep 17 19:15:45.470975 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:45.470917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" event={"ID":"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515","Type":"ContainerDied","Data":"3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0"}
Sep 17 19:15:45.471284 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:45.471252    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:15:45.472002 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:45.471973    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:15:45.494138 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:45.494097    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podStartSLOduration=1.829583585 podStartE2EDuration="23.494085028s" totalImagesPullingTime="17.674099865s" totalInitContainerRuntime="3.990401578s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:45.492911478 +0000 UTC m=+564.620652891" watchObservedRunningTime="2026-09-17 19:15:45.494085028 +0000 UTC m=+564.621826507"
Sep 17 19:15:46.479176 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:46.478796    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:15:46.480426 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:46.480352    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:15:51.485554 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:51.484677    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:15:51.485554 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:51.485358    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:15:51.486229 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:15:51.485905    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:01.485351 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:01.485287    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:16:01.485920 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:01.485850    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:10.608872 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:10.608835    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" event={"ID":"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515","Type":"ContainerStarted","Data":"c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc"}
Sep 17 19:16:10.609345 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:10.608943    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" event={"ID":"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515","Type":"ContainerStarted","Data":"6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac"}
Sep 17 19:16:10.609345 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:10.609170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:16:10.630456 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:10.630407    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podStartSLOduration=5.330678067 podStartE2EDuration="33.630391033s" totalImagesPullingTime="24.357872034s" totalInitContainerRuntime="3.941840932s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:10.629411896 +0000 UTC m=+589.757153307" watchObservedRunningTime="2026-09-17 19:16:10.630391033 +0000 UTC m=+589.758132434"
Sep 17 19:16:11.485218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:11.485172    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:16:11.485642 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:11.485599    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:11.613652 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:11.613625    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:16:11.614808 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:11.614782    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 19:16:12.617718 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:12.617676    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 19:16:17.622023 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:17.621997    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:16:17.622647 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:17.622593    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 19:16:21.485348 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:21.485303    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:16:21.485802 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:21.485782    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:22.933124 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:22.933095    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:16:22.933722 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:22.933699    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:16:27.622634 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:27.622588    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 19:16:31.485525 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:31.485471    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:16:31.485979 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:31.485955    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:37.623388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:37.623307    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 19:16:41.485968 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:41.485929    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:16:41.486540 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:41.486417    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:47.623351 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:47.623310    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 19:16:51.485364 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:51.485314    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:16:51.485774 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:51.485700    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:54.033856 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:54.033810    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:16:54.034255 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:54.034050    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:16:57.623188 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:16:57.623144    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 19:17:04.034325 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:04.034299    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:17:04.034816 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:04.034675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:17:07.622999 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:07.622965    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 19:17:17.434366 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.434333    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"]
Sep 17 19:17:17.434834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.434596    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" containerID="cri-o://36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680" gracePeriod=30
Sep 17 19:17:17.434834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.434665    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy" containerID="cri-o://cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7" gracePeriod=30
Sep 17 19:17:17.434834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.434598    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" containerID="cri-o://074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f" gracePeriod=30
Sep 17 19:17:17.592262 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.592232    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"]
Sep 17 19:17:17.592689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.592667    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa4a1db-762b-4163-8be8-3d788f0fd805" containerName="storage-initializer"
Sep 17 19:17:17.592689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.592685    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa4a1db-762b-4163-8be8-3d788f0fd805" containerName="storage-initializer"
Sep 17 19:17:17.592893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.592700    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa4a1db-762b-4163-8be8-3d788f0fd805" containerName="storage-initializer"
Sep 17 19:17:17.592893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.592705    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa4a1db-762b-4163-8be8-3d788f0fd805" containerName="storage-initializer"
Sep 17 19:17:17.592893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.592761    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa4a1db-762b-4163-8be8-3d788f0fd805" containerName="storage-initializer"
Sep 17 19:17:17.592893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.592871    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa4a1db-762b-4163-8be8-3d788f0fd805" containerName="storage-initializer"
Sep 17 19:17:17.594881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.594866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.599564 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.599545    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 19:17:17.599869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.599852    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 19:17:17.611866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.611836    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"]
Sep 17 19:17:17.623274 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.623247    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 19:17:17.707916 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.707834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.707916 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.707890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2km\" (UniqueName: \"kubernetes.io/projected/c6c6c435-0d51-44d3-a957-529175c479a1-kube-api-access-jd2km\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.708129 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.707935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c6c435-0d51-44d3-a957-529175c479a1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.708129 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.707964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.708129 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.708035    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6c435-0d51-44d3-a957-529175c479a1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.808681 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.808639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.808827 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.808713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2km\" (UniqueName: \"kubernetes.io/projected/c6c6c435-0d51-44d3-a957-529175c479a1-kube-api-access-jd2km\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.808827 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.808743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c6c435-0d51-44d3-a957-529175c479a1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.808827 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.808782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.808827 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.808807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6c435-0d51-44d3-a957-529175c479a1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.809250 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.809221    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c6c435-0d51-44d3-a957-529175c479a1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.809349 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.809333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.809412 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.809391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.811373 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.811357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6c435-0d51-44d3-a957-529175c479a1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.819088 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.819069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2km\" (UniqueName: \"kubernetes.io/projected/c6c6c435-0d51-44d3-a957-529175c479a1-kube-api-access-jd2km\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.905158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.905130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:17.905384 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.905363    2583 generic.go:356] "Generic (PLEG): container finished" podID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerID="cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7" exitCode=2
Sep 17 19:17:17.905442 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:17.905411    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerDied","Data":"cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7"}
Sep 17 19:17:18.182757 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:18.182734    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"]
Sep 17 19:17:18.184993 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:17:18.184961    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c6c435_0d51_44d3_a957_529175c479a1.slice/crio-267e832508960e2dc3adc3dd6af1d97772002c31264bbb2e8dda9de707032bea WatchSource:0}: Error finding container 267e832508960e2dc3adc3dd6af1d97772002c31264bbb2e8dda9de707032bea: Status 404 returned error can't find the container with id 267e832508960e2dc3adc3dd6af1d97772002c31264bbb2e8dda9de707032bea
Sep 17 19:17:18.911461 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:18.911429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerStarted","Data":"772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0"}
Sep 17 19:17:18.911897 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:18.911498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerStarted","Data":"267e832508960e2dc3adc3dd6af1d97772002c31264bbb2e8dda9de707032bea"}
Sep 17 19:17:21.479682 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:21.479644    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 17 19:17:22.929824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:22.929793    2583 generic.go:356] "Generic (PLEG): container finished" podID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerID="36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680" exitCode=0
Sep 17 19:17:22.930246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:22.929850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerDied","Data":"36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680"}
Sep 17 19:17:22.937192 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:22.937159    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6c6c435-0d51-44d3-a957-529175c479a1" containerID="772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0" exitCode=0
Sep 17 19:17:22.937331 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:22.937208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerDied","Data":"772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0"}
Sep 17 19:17:23.946357 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:23.946324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerStarted","Data":"be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab"}
Sep 17 19:17:23.946838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:23.946392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerStarted","Data":"e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08"}
Sep 17 19:17:23.946838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:23.946430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerStarted","Data":"3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134"}
Sep 17 19:17:23.946838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:23.946809    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:23.947013 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:23.946868    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:23.947013 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:23.946904    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:23.948414 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:23.948381    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:17:23.949109 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:23.949078    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:23.968694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:23.968648    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podStartSLOduration=3.131940361 podStartE2EDuration="6.968631774s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.836691413s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:17:23.966939491 +0000 UTC m=+663.094680913" watchObservedRunningTime="2026-09-17 19:17:23.968631774 +0000 UTC m=+663.096373175"
Sep 17 19:17:24.034300 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:24.034265    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:24.034770 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:24.034750    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:17:24.953731 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:24.953676    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:17:24.954183 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:24.954058    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:26.479223 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:26.479180    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 17 19:17:27.623777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:27.623748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:17:29.957572 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:29.957546    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:17:29.958109 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:29.958083    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:17:29.958443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:29.958423    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:31.479939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:31.479890    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 17 19:17:31.480330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:31.480015    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:17:34.034061 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:34.034027    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:34.035282 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:34.035253    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:17:36.479244 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:36.479204    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 17 19:17:37.557328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:37.557287    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"]
Sep 17 19:17:37.557763 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:37.557524    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container" containerID="cri-o://6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac" gracePeriod=30
Sep 17 19:17:37.557763 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:37.557626    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kube-rbac-proxy" containerID="cri-o://c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc" gracePeriod=30
Sep 17 19:17:37.618564 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:37.618532    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 17 19:17:38.011307 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:38.011275    2583 generic.go:356] "Generic (PLEG): container finished" podID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerID="c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc" exitCode=2
Sep 17 19:17:38.011464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:38.011349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" event={"ID":"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515","Type":"ContainerDied","Data":"c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc"}
Sep 17 19:17:39.958435 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:39.958396    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:17:39.958950 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:39.958928    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:41.479459 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:41.479422    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 17 19:17:42.292922 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.292898    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:17:42.412274 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.412190    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kserve-provision-location\") pod \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") "
Sep 17 19:17:42.412274 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.412232    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-cabundle-cert\") pod \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") "
Sep 17 19:17:42.412464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.412309    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-proxy-tls\") pod \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") "
Sep 17 19:17:42.412464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.412343    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8dqj\" (UniqueName: \"kubernetes.io/projected/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-api-access-q8dqj\") pod \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") "
Sep 17 19:17:42.412464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.412378    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-rbac-proxy-sar-config\") pod \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\" (UID: \"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515\") "
Sep 17 19:17:42.412646 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.412566    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" (UID: "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:17:42.412646 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.412597    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" (UID: "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:17:42.412862 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.412839    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" (UID: "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:17:42.414648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.414604    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" (UID: "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:17:42.414648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.414633    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-api-access-q8dqj" (OuterVolumeSpecName: "kube-api-access-q8dqj") pod "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" (UID: "fca96e10-efbc-4eea-b6e4-7ffa8a7c9515"). InnerVolumeSpecName "kube-api-access-q8dqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:17:42.513117 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.513093    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:42.513117 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.513116    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:42.513441 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.513127    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:42.513441 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.513136    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8dqj\" (UniqueName: \"kubernetes.io/projected/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-api-access-q8dqj\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:42.513441 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.513144    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:42.744450 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.744411    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"]
Sep 17 19:17:42.744853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.744836    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container"
Sep 17 19:17:42.744951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.744856    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container"
Sep 17 19:17:42.744951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.744867    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kube-rbac-proxy"
Sep 17 19:17:42.744951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.744876    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kube-rbac-proxy"
Sep 17 19:17:42.744951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.744914    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="storage-initializer"
Sep 17 19:17:42.744951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.744924    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="storage-initializer"
Sep 17 19:17:42.745196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.745013    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kube-rbac-proxy"
Sep 17 19:17:42.745196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.745024    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerName="kserve-container"
Sep 17 19:17:42.747328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.747307    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.749428 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.749405    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 19:17:42.749546 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.749523    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 19:17:42.756925 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.756904    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"]
Sep 17 19:17:42.814478 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.814452    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.814593 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.814482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c872253-e8c4-4cdc-a548-4f738febbc97-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.814593 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.814503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.814705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.814587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6wcw\" (UniqueName: \"kubernetes.io/projected/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-api-access-k6wcw\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.814705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.814680    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c872253-e8c4-4cdc-a548-4f738febbc97-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.915040 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.915006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c872253-e8c4-4cdc-a548-4f738febbc97-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.915185 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.915057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.915185 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.915169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c872253-e8c4-4cdc-a548-4f738febbc97-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.915307 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.915214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.915307 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.915259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6wcw\" (UniqueName: \"kubernetes.io/projected/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-api-access-k6wcw\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.915563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.915542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c872253-e8c4-4cdc-a548-4f738febbc97-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.915768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.915749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.915872 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.915853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.917587 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.917569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c872253-e8c4-4cdc-a548-4f738febbc97-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:42.923137 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:42.923120    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6wcw\" (UniqueName: \"kubernetes.io/projected/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-api-access-k6wcw\") pod \"isvc-predictive-xgboost-predictor-695f978978-gltsd\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:43.039406 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.039332    2583 generic.go:356] "Generic (PLEG): container finished" podID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" containerID="6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac" exitCode=0
Sep 17 19:17:43.039546 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.039443    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"
Sep 17 19:17:43.041389 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.041366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" event={"ID":"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515","Type":"ContainerDied","Data":"6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac"}
Sep 17 19:17:43.041488 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.041411    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b" event={"ID":"fca96e10-efbc-4eea-b6e4-7ffa8a7c9515","Type":"ContainerDied","Data":"d43b2e3d675ba23a0b08f5af4fa167f483ed9a540157d979a6b90e8ea4786b1e"}
Sep 17 19:17:43.041488 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.041451    2583 scope.go:117] "RemoveContainer" containerID="c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc"
Sep 17 19:17:43.051837 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.051819    2583 scope.go:117] "RemoveContainer" containerID="6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac"
Sep 17 19:17:43.058151 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.058129    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:43.059713 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.059691    2583 scope.go:117] "RemoveContainer" containerID="3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0"
Sep 17 19:17:43.064497 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.064474    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"]
Sep 17 19:17:43.069385 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.069363    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-9mf4b"]
Sep 17 19:17:43.070462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.070450    2583 scope.go:117] "RemoveContainer" containerID="c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc"
Sep 17 19:17:43.070794 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:17:43.070761    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc\": container with ID starting with c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc not found: ID does not exist" containerID="c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc"
Sep 17 19:17:43.070881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.070792    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc"} err="failed to get container status \"c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc\": rpc error: code = NotFound desc = could not find container \"c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc\": container with ID starting with c99f0cc8f5e7a82914f89baab8408ab9cc10774c41a3ff73854730aa1dcae0fc not found: ID does not exist"
Sep 17 19:17:43.070881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.070823    2583 scope.go:117] "RemoveContainer" containerID="6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac"
Sep 17 19:17:43.071134 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:17:43.071100    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac\": container with ID starting with 6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac not found: ID does not exist" containerID="6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac"
Sep 17 19:17:43.071173 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.071144    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac"} err="failed to get container status \"6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac\": rpc error: code = NotFound desc = could not find container \"6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac\": container with ID starting with 6a2e11641b59398bb974e73d666d6190149c5140529a46874ecdbf56f14697ac not found: ID does not exist"
Sep 17 19:17:43.071173 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.071166    2583 scope.go:117] "RemoveContainer" containerID="3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0"
Sep 17 19:17:43.071422 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:17:43.071405    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0\": container with ID starting with 3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0 not found: ID does not exist" containerID="3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0"
Sep 17 19:17:43.071464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.071427    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0"} err="failed to get container status \"3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0\": rpc error: code = NotFound desc = could not find container \"3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0\": container with ID starting with 3f9052320b31eacdb2234e2cc735ebe148eec7614f8af2e7f7e57cb9dce4f8f0 not found: ID does not exist"
Sep 17 19:17:43.336089 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:43.336068    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"]
Sep 17 19:17:43.338238 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:17:43.338211    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c872253_e8c4_4cdc_a548_4f738febbc97.slice/crio-1fb4e77387213f4feae3ea21f0ec2964fe8da54da6a2a8a9cf9e96352402a952 WatchSource:0}: Error finding container 1fb4e77387213f4feae3ea21f0ec2964fe8da54da6a2a8a9cf9e96352402a952: Status 404 returned error can't find the container with id 1fb4e77387213f4feae3ea21f0ec2964fe8da54da6a2a8a9cf9e96352402a952
Sep 17 19:17:44.034812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:44.034770    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 19:17:44.035192 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:44.034766    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:44.035192 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:44.034923    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:17:44.035192 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:44.035039    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:17:44.047293 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:44.047261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" event={"ID":"5c872253-e8c4-4cdc-a548-4f738febbc97","Type":"ContainerStarted","Data":"0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee"}
Sep 17 19:17:44.047428 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:44.047329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" event={"ID":"5c872253-e8c4-4cdc-a548-4f738febbc97","Type":"ContainerStarted","Data":"1fb4e77387213f4feae3ea21f0ec2964fe8da54da6a2a8a9cf9e96352402a952"}
Sep 17 19:17:45.038594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:45.038559    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fca96e10-efbc-4eea-b6e4-7ffa8a7c9515" path="/var/lib/kubelet/pods/fca96e10-efbc-4eea-b6e4-7ffa8a7c9515/volumes"
Sep 17 19:17:46.479373 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:46.479329    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 17 19:17:47.580168 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.580141    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:17:47.658676 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.658643    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-cabundle-cert\") pod \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") "
Sep 17 19:17:47.658676 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.658683    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kserve-provision-location\") pod \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") "
Sep 17 19:17:47.658893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.658737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcsgq\" (UniqueName: \"kubernetes.io/projected/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-api-access-zcsgq\") pod \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") "
Sep 17 19:17:47.658893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.658768    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-proxy-tls\") pod \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") "
Sep 17 19:17:47.658893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.658819    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-rbac-proxy-sar-config\") pod \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\" (UID: \"12da2abd-f0e8-434c-a44e-fe52ddc4c28d\") "
Sep 17 19:17:47.659053 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.659032    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12da2abd-f0e8-434c-a44e-fe52ddc4c28d" (UID: "12da2abd-f0e8-434c-a44e-fe52ddc4c28d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:17:47.659112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.659066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12da2abd-f0e8-434c-a44e-fe52ddc4c28d" (UID: "12da2abd-f0e8-434c-a44e-fe52ddc4c28d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:17:47.659260 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.659237    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12da2abd-f0e8-434c-a44e-fe52ddc4c28d" (UID: "12da2abd-f0e8-434c-a44e-fe52ddc4c28d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:17:47.660998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.660971    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12da2abd-f0e8-434c-a44e-fe52ddc4c28d" (UID: "12da2abd-f0e8-434c-a44e-fe52ddc4c28d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:17:47.661102 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.661003    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-api-access-zcsgq" (OuterVolumeSpecName: "kube-api-access-zcsgq") pod "12da2abd-f0e8-434c-a44e-fe52ddc4c28d" (UID: "12da2abd-f0e8-434c-a44e-fe52ddc4c28d"). InnerVolumeSpecName "kube-api-access-zcsgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:17:47.759824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.759750    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:47.759824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.759776    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:47.759824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.759786    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcsgq\" (UniqueName: \"kubernetes.io/projected/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-api-access-zcsgq\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:47.759824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.759796    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:47.759824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:47.759804    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da2abd-f0e8-434c-a44e-fe52ddc4c28d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:17:48.067484 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.067400    2583 generic.go:356] "Generic (PLEG): container finished" podID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerID="074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f" exitCode=0
Sep 17 19:17:48.067484 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.067480    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"
Sep 17 19:17:48.067739 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.067481    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerDied","Data":"074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f"}
Sep 17 19:17:48.067739 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.067661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8" event={"ID":"12da2abd-f0e8-434c-a44e-fe52ddc4c28d","Type":"ContainerDied","Data":"0bbed9bd6629f5788a9cc9f3d87bcfeab31549effda1ed5a6d9725eaadca2583"}
Sep 17 19:17:48.067739 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.067705    2583 scope.go:117] "RemoveContainer" containerID="074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f"
Sep 17 19:17:48.069440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.069421    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerID="0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee" exitCode=0
Sep 17 19:17:48.069544 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.069462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" event={"ID":"5c872253-e8c4-4cdc-a548-4f738febbc97","Type":"ContainerDied","Data":"0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee"}
Sep 17 19:17:48.076243 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.076215    2583 scope.go:117] "RemoveContainer" containerID="cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7"
Sep 17 19:17:48.085200 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.085185    2583 scope.go:117] "RemoveContainer" containerID="36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680"
Sep 17 19:17:48.097923 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.097902    2583 scope.go:117] "RemoveContainer" containerID="61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172"
Sep 17 19:17:48.104056 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.104028    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"]
Sep 17 19:17:48.107688 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.107666    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-6bmh8"]
Sep 17 19:17:48.111282 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.111263    2583 scope.go:117] "RemoveContainer" containerID="074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f"
Sep 17 19:17:48.111635 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:17:48.111603    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f\": container with ID starting with 074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f not found: ID does not exist" containerID="074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f"
Sep 17 19:17:48.111702 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.111645    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f"} err="failed to get container status \"074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f\": rpc error: code = NotFound desc = could not find container \"074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f\": container with ID starting with 074deb81cfa8545a305f7171f62f7ef4974416e8e18f460fab53782fb87bdb1f not found: ID does not exist"
Sep 17 19:17:48.111702 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.111665    2583 scope.go:117] "RemoveContainer" containerID="cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7"
Sep 17 19:17:48.111996 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:17:48.111977    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7\": container with ID starting with cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7 not found: ID does not exist" containerID="cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7"
Sep 17 19:17:48.112055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.112001    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7"} err="failed to get container status \"cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7\": rpc error: code = NotFound desc = could not find container \"cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7\": container with ID starting with cd954bca301485dbd4afd5e5d9a8ab3b50f27485d779d8c018020c9566c7d0e7 not found: ID does not exist"
Sep 17 19:17:48.112055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.112017    2583 scope.go:117] "RemoveContainer" containerID="36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680"
Sep 17 19:17:48.112297 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:17:48.112269    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680\": container with ID starting with 36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680 not found: ID does not exist" containerID="36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680"
Sep 17 19:17:48.112354 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.112300    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680"} err="failed to get container status \"36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680\": rpc error: code = NotFound desc = could not find container \"36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680\": container with ID starting with 36eb997602a0fcd3a82c57d16d02fe602a19e4a8757b388e30f8a5d034930680 not found: ID does not exist"
Sep 17 19:17:48.112354 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.112314    2583 scope.go:117] "RemoveContainer" containerID="61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172"
Sep 17 19:17:48.112540 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:17:48.112521    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172\": container with ID starting with 61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172 not found: ID does not exist" containerID="61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172"
Sep 17 19:17:48.112593 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:48.112550    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172"} err="failed to get container status \"61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172\": rpc error: code = NotFound desc = could not find container \"61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172\": container with ID starting with 61786aa5b3635a59f8c8564e38c9e4dce063606544a50072f3a493ae21821172 not found: ID does not exist"
Sep 17 19:17:49.039691 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:49.039656    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" path="/var/lib/kubelet/pods/12da2abd-f0e8-434c-a44e-fe52ddc4c28d/volumes"
Sep 17 19:17:49.076920 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:49.076886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" event={"ID":"5c872253-e8c4-4cdc-a548-4f738febbc97","Type":"ContainerStarted","Data":"47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30"}
Sep 17 19:17:49.077055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:49.076951    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" event={"ID":"5c872253-e8c4-4cdc-a548-4f738febbc97","Type":"ContainerStarted","Data":"563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6"}
Sep 17 19:17:49.077159 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:49.077143    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:49.096708 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:49.096668    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podStartSLOduration=3.428319617 podStartE2EDuration="7.096655328s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.668335711s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:17:49.096008406 +0000 UTC m=+688.223749840" watchObservedRunningTime="2026-09-17 19:17:49.096655328 +0000 UTC m=+688.224396727"
Sep 17 19:17:49.958077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:49.958039    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:17:49.958410 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:49.958387    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:17:50.081469 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:50.081436    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:50.082658 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:50.082610    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:17:51.085693 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:51.085652    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:17:56.090783 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:56.090755    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:17:56.091253 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:56.091221    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:17:59.958729 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:59.958682    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:17:59.959209 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:17:59.959184    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:18:06.091381 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:06.091279    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:18:09.958796 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:09.958752    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:18:09.959247 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:09.959166    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:18:16.092120 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:16.092081    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:18:19.958511 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:19.958462    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:18:19.959082 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:19.959059    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:18:26.091830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:26.091786    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:18:29.958895 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:29.958846    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:18:29.959357 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:29.959336    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:18:36.091165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:36.091127    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:18:39.033998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:39.033959    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:18:39.034469 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:39.034208    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:18:46.091996 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:46.091946    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:18:49.038700 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:49.038671    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:18:49.039061 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:49.038756    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:18:56.091775 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:18:56.091728    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:19:03.065374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.065342    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"]
Sep 17 19:19:03.065890 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.065663    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" containerID="cri-o://3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134" gracePeriod=30
Sep 17 19:19:03.065890 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.065740    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy" containerID="cri-o://e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08" gracePeriod=30
Sep 17 19:19:03.066059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.066034    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" containerID="cri-o://be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab" gracePeriod=30
Sep 17 19:19:03.208196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208167    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"]
Sep 17 19:19:03.208521 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208509    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="storage-initializer"
Sep 17 19:19:03.208565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208523    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="storage-initializer"
Sep 17 19:19:03.208565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208531    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container"
Sep 17 19:19:03.208565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208537    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container"
Sep 17 19:19:03.208687 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208573    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent"
Sep 17 19:19:03.208687 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208579    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent"
Sep 17 19:19:03.208687 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208586    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy"
Sep 17 19:19:03.208687 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208591    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy"
Sep 17 19:19:03.208687 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208667    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kube-rbac-proxy"
Sep 17 19:19:03.208687 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208678    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="kserve-container"
Sep 17 19:19:03.208687 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.208685    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="12da2abd-f0e8-434c-a44e-fe52ddc4c28d" containerName="agent"
Sep 17 19:19:03.210723 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.210709    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.213364 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.213344    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 19:19:03.213460 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.213350    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 19:19:03.265332 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.265305    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"]
Sep 17 19:19:03.307818 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.307792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cac9a39-7017-4bd5-b81e-7159ea25710d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.307967 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.307855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.308015 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.307965    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nckcg\" (UniqueName: \"kubernetes.io/projected/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-api-access-nckcg\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.394002 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.393931    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6c6c435-0d51-44d3-a957-529175c479a1" containerID="e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08" exitCode=2
Sep 17 19:19:03.394133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.394007    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerDied","Data":"e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08"}
Sep 17 19:19:03.408432 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.408408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nckcg\" (UniqueName: \"kubernetes.io/projected/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-api-access-nckcg\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.408522 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.408448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cac9a39-7017-4bd5-b81e-7159ea25710d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.408522 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.408480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.408658 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:03.408581    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 17 19:19:03.408719 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:03.408678    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cac9a39-7017-4bd5-b81e-7159ea25710d-proxy-tls podName:3cac9a39-7017-4bd5-b81e-7159ea25710d nodeName:}" failed. No retries permitted until 2026-09-17 19:19:03.908655791 +0000 UTC m=+763.036397170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3cac9a39-7017-4bd5-b81e-7159ea25710d-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-pq2fl" (UID: "3cac9a39-7017-4bd5-b81e-7159ea25710d") : secret "message-dumper-predictor-serving-cert" not found
Sep 17 19:19:03.409131 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.409112    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.416965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.416947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nckcg\" (UniqueName: \"kubernetes.io/projected/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-api-access-nckcg\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.912754 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.912719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cac9a39-7017-4bd5-b81e-7159ea25710d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:03.915241 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:03.915216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cac9a39-7017-4bd5-b81e-7159ea25710d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-pq2fl\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:04.034351 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:04.034322    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:19:04.120412 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:04.120384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:04.389989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:04.389965    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"]
Sep 17 19:19:04.392600 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:19:04.392569    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cac9a39_7017_4bd5_b81e_7159ea25710d.slice/crio-a1f0605451dc578a60b38d06efb3b3bc99ff916da656885a7cefb40ae2420c65 WatchSource:0}: Error finding container a1f0605451dc578a60b38d06efb3b3bc99ff916da656885a7cefb40ae2420c65: Status 404 returned error can't find the container with id a1f0605451dc578a60b38d06efb3b3bc99ff916da656885a7cefb40ae2420c65
Sep 17 19:19:04.394581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:04.394561    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:19:04.400012 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:04.399945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl" event={"ID":"3cac9a39-7017-4bd5-b81e-7159ea25710d","Type":"ContainerStarted","Data":"a1f0605451dc578a60b38d06efb3b3bc99ff916da656885a7cefb40ae2420c65"}
Sep 17 19:19:04.953768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:04.953730    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 17 19:19:06.410807 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:06.410769    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl" event={"ID":"3cac9a39-7017-4bd5-b81e-7159ea25710d","Type":"ContainerStarted","Data":"8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546"}
Sep 17 19:19:06.411222 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:06.410834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl" event={"ID":"3cac9a39-7017-4bd5-b81e-7159ea25710d","Type":"ContainerStarted","Data":"84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248"}
Sep 17 19:19:06.411222 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:06.410859    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:06.411222 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:06.410885    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:06.412790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:06.412768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:06.430812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:06.430748    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl" podStartSLOduration=2.233879447 podStartE2EDuration="3.430729787s" totalImagesPullingTime="1.19685034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:06.430485471 +0000 UTC m=+765.558226874" watchObservedRunningTime="2026-09-17 19:19:06.430729787 +0000 UTC m=+765.558471189"
Sep 17 19:19:08.421374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:08.421344    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6c6c435-0d51-44d3-a957-529175c479a1" containerID="3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134" exitCode=0
Sep 17 19:19:08.421824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:08.421423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerDied","Data":"3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134"}
Sep 17 19:19:09.034083 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:09.034038    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:19:09.034466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:09.034440    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:09.954270 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:09.954230    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 17 19:19:13.256532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:13.256501    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"]
Sep 17 19:19:13.257019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:13.256804    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" containerID="cri-o://563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6" gracePeriod=30
Sep 17 19:19:13.257019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:13.256884    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kube-rbac-proxy" containerID="cri-o://47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30" gracePeriod=30
Sep 17 19:19:13.425641 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:13.425591    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:19:13.445202 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:13.445170    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerID="47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30" exitCode=2
Sep 17 19:19:13.445373 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:13.445250    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" event={"ID":"5c872253-e8c4-4cdc-a548-4f738febbc97","Type":"ContainerDied","Data":"47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30"}
Sep 17 19:19:14.033718 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:14.033678    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 19:19:14.954274 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:14.954228    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 17 19:19:14.954690 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:14.954367    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:19:16.086034 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:16.085991    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 17 19:19:18.296719 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.296695    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:19:18.435510 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.435481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6wcw\" (UniqueName: \"kubernetes.io/projected/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-api-access-k6wcw\") pod \"5c872253-e8c4-4cdc-a548-4f738febbc97\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") "
Sep 17 19:19:18.435697 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.435529    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c872253-e8c4-4cdc-a548-4f738febbc97-proxy-tls\") pod \"5c872253-e8c4-4cdc-a548-4f738febbc97\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") "
Sep 17 19:19:18.435697 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.435556    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-cabundle-cert\") pod \"5c872253-e8c4-4cdc-a548-4f738febbc97\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") "
Sep 17 19:19:18.435697 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.435676    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-rbac-proxy-sar-config\") pod \"5c872253-e8c4-4cdc-a548-4f738febbc97\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") "
Sep 17 19:19:18.435829 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.435718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c872253-e8c4-4cdc-a548-4f738febbc97-kserve-provision-location\") pod \"5c872253-e8c4-4cdc-a548-4f738febbc97\" (UID: \"5c872253-e8c4-4cdc-a548-4f738febbc97\") "
Sep 17 19:19:18.435959 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.435932    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c872253-e8c4-4cdc-a548-4f738febbc97" (UID: "5c872253-e8c4-4cdc-a548-4f738febbc97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:18.436116 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.436096    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c872253-e8c4-4cdc-a548-4f738febbc97" (UID: "5c872253-e8c4-4cdc-a548-4f738febbc97"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:18.436187 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.436109    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c872253-e8c4-4cdc-a548-4f738febbc97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c872253-e8c4-4cdc-a548-4f738febbc97" (UID: "5c872253-e8c4-4cdc-a548-4f738febbc97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:18.436187 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.436161    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:18.437994 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.437965    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-api-access-k6wcw" (OuterVolumeSpecName: "kube-api-access-k6wcw") pod "5c872253-e8c4-4cdc-a548-4f738febbc97" (UID: "5c872253-e8c4-4cdc-a548-4f738febbc97"). InnerVolumeSpecName "kube-api-access-k6wcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:18.437994 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.437985    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c872253-e8c4-4cdc-a548-4f738febbc97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c872253-e8c4-4cdc-a548-4f738febbc97" (UID: "5c872253-e8c4-4cdc-a548-4f738febbc97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:18.468769 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.468745    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerID="563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6" exitCode=0
Sep 17 19:19:18.468869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.468803    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"
Sep 17 19:19:18.468869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.468807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" event={"ID":"5c872253-e8c4-4cdc-a548-4f738febbc97","Type":"ContainerDied","Data":"563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6"}
Sep 17 19:19:18.469009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.468869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd" event={"ID":"5c872253-e8c4-4cdc-a548-4f738febbc97","Type":"ContainerDied","Data":"1fb4e77387213f4feae3ea21f0ec2964fe8da54da6a2a8a9cf9e96352402a952"}
Sep 17 19:19:18.469009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.468908    2583 scope.go:117] "RemoveContainer" containerID="47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30"
Sep 17 19:19:18.477694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.477673    2583 scope.go:117] "RemoveContainer" containerID="563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6"
Sep 17 19:19:18.485464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.485444    2583 scope.go:117] "RemoveContainer" containerID="0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee"
Sep 17 19:19:18.490603 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.490583    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"]
Sep 17 19:19:18.490943 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.490930    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="storage-initializer"
Sep 17 19:19:18.490984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.490945    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="storage-initializer"
Sep 17 19:19:18.490984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.490955    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container"
Sep 17 19:19:18.490984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.490960    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container"
Sep 17 19:19:18.490984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.490968    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kube-rbac-proxy"
Sep 17 19:19:18.490984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.490974    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kube-rbac-proxy"
Sep 17 19:19:18.491131 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.491031    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kube-rbac-proxy"
Sep 17 19:19:18.491131 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.491040    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" containerName="kserve-container"
Sep 17 19:19:18.494179 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.494161    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.494504 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.494484    2583 scope.go:117] "RemoveContainer" containerID="47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30"
Sep 17 19:19:18.494837 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:18.494818    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30\": container with ID starting with 47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30 not found: ID does not exist" containerID="47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30"
Sep 17 19:19:18.494892 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.494845    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30"} err="failed to get container status \"47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30\": rpc error: code = NotFound desc = could not find container \"47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30\": container with ID starting with 47a57a726099200590c1e1ba27504246a7fd88a7fb28840d619f228080091a30 not found: ID does not exist"
Sep 17 19:19:18.494892 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.494862    2583 scope.go:117] "RemoveContainer" containerID="563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6"
Sep 17 19:19:18.495126 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:18.495110    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6\": container with ID starting with 563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6 not found: ID does not exist" containerID="563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6"
Sep 17 19:19:18.495187 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.495135    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6"} err="failed to get container status \"563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6\": rpc error: code = NotFound desc = could not find container \"563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6\": container with ID starting with 563d66aff9555cc9486fc635e529bdae7991580aadf735e883b2dc84f5076de6 not found: ID does not exist"
Sep 17 19:19:18.495187 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.495156    2583 scope.go:117] "RemoveContainer" containerID="0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee"
Sep 17 19:19:18.495444 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:18.495427    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee\": container with ID starting with 0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee not found: ID does not exist" containerID="0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee"
Sep 17 19:19:18.495485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.495449    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee"} err="failed to get container status \"0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee\": rpc error: code = NotFound desc = could not find container \"0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee\": container with ID starting with 0077fb8c5532713eb6591b9ae6adfbd22a9f236010522916d8a8569feadc7eee not found: ID does not exist"
Sep 17 19:19:18.499192 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.499175    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 19:19:18.499759 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.499744    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 19:19:18.514875 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.514851    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"]
Sep 17 19:19:18.522965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.522948    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"]
Sep 17 19:19:18.527403 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.527381    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gltsd"]
Sep 17 19:19:18.536711 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.536692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.536796 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.536728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.536796 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.536760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.536871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.536821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22n6g\" (UniqueName: \"kubernetes.io/projected/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-api-access-22n6g\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.536902 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.536865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.536969 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.536955    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6wcw\" (UniqueName: \"kubernetes.io/projected/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-api-access-k6wcw\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:18.537004 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.536971    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c872253-e8c4-4cdc-a548-4f738febbc97-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:18.537004 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.536982    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c872253-e8c4-4cdc-a548-4f738febbc97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:18.537004 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.536991    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c872253-e8c4-4cdc-a548-4f738febbc97-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:18.637844 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.637810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.638021 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.637865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.638021 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.637896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.638021 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.637928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22n6g\" (UniqueName: \"kubernetes.io/projected/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-api-access-22n6g\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.638021 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:18.637954    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 19:19:18.638021 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.637972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.638244 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:18.638024    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-proxy-tls podName:8bafcb2b-52ad-4671-9d62-7a62199d9e8b nodeName:}" failed. No retries permitted until 2026-09-17 19:19:19.138002792 +0000 UTC m=+778.265744174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" (UID: "8bafcb2b-52ad-4671-9d62-7a62199d9e8b") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 19:19:18.638428 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.638407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.638697 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.638676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.638894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.638877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:18.646636 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:18.646594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22n6g\" (UniqueName: \"kubernetes.io/projected/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-api-access-22n6g\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:19.033984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:19.033943    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:19:19.034275 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:19.034243    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:19.044715 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:19.044684    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c872253-e8c4-4cdc-a548-4f738febbc97" path="/var/lib/kubelet/pods/5c872253-e8c4-4cdc-a548-4f738febbc97/volumes"
Sep 17 19:19:19.142237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:19.142201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:19.144730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:19.144712    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:19.404468 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:19.404378    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:19.678790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:19.678754    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"]
Sep 17 19:19:19.681077 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:19:19.681052    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bafcb2b_52ad_4671_9d62_7a62199d9e8b.slice/crio-be955b19b01cc72c194206d6ee850874fb478b3b57f7a4d2250e3f44bb30509c WatchSource:0}: Error finding container be955b19b01cc72c194206d6ee850874fb478b3b57f7a4d2250e3f44bb30509c: Status 404 returned error can't find the container with id be955b19b01cc72c194206d6ee850874fb478b3b57f7a4d2250e3f44bb30509c
Sep 17 19:19:19.954810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:19.954718    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 17 19:19:20.482191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:20.482160    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" event={"ID":"8bafcb2b-52ad-4671-9d62-7a62199d9e8b","Type":"ContainerStarted","Data":"4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4"}
Sep 17 19:19:20.482587 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:20.482214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" event={"ID":"8bafcb2b-52ad-4671-9d62-7a62199d9e8b","Type":"ContainerStarted","Data":"be955b19b01cc72c194206d6ee850874fb478b3b57f7a4d2250e3f44bb30509c"}
Sep 17 19:19:23.251133 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.251056    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"]
Sep 17 19:19:23.254459 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.254442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.256793 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.256774    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 19:19:23.256884 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.256839    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 19:19:23.264946 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.264924    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"]
Sep 17 19:19:23.379111 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.379083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-cabundle-cert\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.379277 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.379139    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e50d729-a900-4852-b1c4-594aea5acd8e-kserve-provision-location\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.379277 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.379182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5bs\" (UniqueName: \"kubernetes.io/projected/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-api-access-ts5bs\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.379277 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.379214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.379277 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.379245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e50d729-a900-4852-b1c4-594aea5acd8e-proxy-tls\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.480581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.480553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-cabundle-cert\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.480760 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.480635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e50d729-a900-4852-b1c4-594aea5acd8e-kserve-provision-location\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.480760 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.480666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5bs\" (UniqueName: \"kubernetes.io/projected/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-api-access-ts5bs\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.480760 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.480701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.480760 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.480739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e50d729-a900-4852-b1c4-594aea5acd8e-proxy-tls\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.481061 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.481038    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e50d729-a900-4852-b1c4-594aea5acd8e-kserve-provision-location\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.481208 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.481190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-cabundle-cert\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.481363 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.481346    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.483265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.483244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e50d729-a900-4852-b1c4-594aea5acd8e-proxy-tls\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.489288 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.489268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5bs\" (UniqueName: \"kubernetes.io/projected/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-api-access-ts5bs\") pod \"isvc-logger-predictor-8559c4556c-cjhd6\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.564598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.564509    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:23.843158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:23.842991    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"]
Sep 17 19:19:23.871483 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:19:23.871455    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e50d729_a900_4852_b1c4_594aea5acd8e.slice/crio-9818a9f3f081db4881b7fe7c81a86c360ead5b699f1c136b340a617b6a6987ca WatchSource:0}: Error finding container 9818a9f3f081db4881b7fe7c81a86c360ead5b699f1c136b340a617b6a6987ca: Status 404 returned error can't find the container with id 9818a9f3f081db4881b7fe7c81a86c360ead5b699f1c136b340a617b6a6987ca
Sep 17 19:19:24.500904 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:24.500867    2583 generic.go:356] "Generic (PLEG): container finished" podID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerID="4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4" exitCode=0
Sep 17 19:19:24.501374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:24.500952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" event={"ID":"8bafcb2b-52ad-4671-9d62-7a62199d9e8b","Type":"ContainerDied","Data":"4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4"}
Sep 17 19:19:24.502750 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:24.502709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerStarted","Data":"1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3"}
Sep 17 19:19:24.502854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:24.502784    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerStarted","Data":"9818a9f3f081db4881b7fe7c81a86c360ead5b699f1c136b340a617b6a6987ca"}
Sep 17 19:19:24.954061 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:24.954024    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 17 19:19:25.510188 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:25.510149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" event={"ID":"8bafcb2b-52ad-4671-9d62-7a62199d9e8b","Type":"ContainerStarted","Data":"1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec"}
Sep 17 19:19:25.510660 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:25.510207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" event={"ID":"8bafcb2b-52ad-4671-9d62-7a62199d9e8b","Type":"ContainerStarted","Data":"9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec"}
Sep 17 19:19:25.510660 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:25.510450    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:25.530755 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:25.530707    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podStartSLOduration=3.457944584 podStartE2EDuration="7.53069214s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.072747556s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:25.529184404 +0000 UTC m=+784.656925806" watchObservedRunningTime="2026-09-17 19:19:25.53069214 +0000 UTC m=+784.658433540"
Sep 17 19:19:26.514713 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:26.514683    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:26.515926 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:26.515898    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:19:27.519691 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:27.519656    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:19:28.526216 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:28.526183    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerID="1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3" exitCode=0
Sep 17 19:19:28.526582 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:28.526257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerDied","Data":"1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3"}
Sep 17 19:19:29.033981 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.033943    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:5000: connect: connection refused"
Sep 17 19:19:29.034289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.034265    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:29.038568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.038544    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:19:29.038716 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.038675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:19:29.534319 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.534287    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerStarted","Data":"6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e"}
Sep 17 19:19:29.534695 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.534357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerStarted","Data":"48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222"}
Sep 17 19:19:29.534695 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.534381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerStarted","Data":"eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17"}
Sep 17 19:19:29.534695 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.534672    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:29.534830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.534819    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:29.536149 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.536125    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:29.556428 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.556345    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podStartSLOduration=2.6432837559999998 podStartE2EDuration="6.556331732s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.913047976s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:29.554648006 +0000 UTC m=+788.682389407" watchObservedRunningTime="2026-09-17 19:19:29.556331732 +0000 UTC m=+788.684073132"
Sep 17 19:19:29.954414 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:29.954381    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 17 19:19:30.538751 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:30.538711    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:30.539182 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:30.538851    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:30.540037 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:30.540012    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:31.543927 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:31.543882    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:31.544420 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:31.544391    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:32.524188 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:32.524120    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:19:32.524610 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:32.524586    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:19:33.209399 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.209377    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:19:33.368055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.367976    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6c435-0d51-44d3-a957-529175c479a1-proxy-tls\") pod \"c6c6c435-0d51-44d3-a957-529175c479a1\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") "
Sep 17 19:19:33.368055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.368016    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c6c435-0d51-44d3-a957-529175c479a1-kserve-provision-location\") pod \"c6c6c435-0d51-44d3-a957-529175c479a1\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") "
Sep 17 19:19:33.368055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.368046    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-cabundle-cert\") pod \"c6c6c435-0d51-44d3-a957-529175c479a1\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") "
Sep 17 19:19:33.368265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.368077    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd2km\" (UniqueName: \"kubernetes.io/projected/c6c6c435-0d51-44d3-a957-529175c479a1-kube-api-access-jd2km\") pod \"c6c6c435-0d51-44d3-a957-529175c479a1\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") "
Sep 17 19:19:33.368265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.368110    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-kube-rbac-proxy-sar-config\") pod \"c6c6c435-0d51-44d3-a957-529175c479a1\" (UID: \"c6c6c435-0d51-44d3-a957-529175c479a1\") "
Sep 17 19:19:33.368334 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.368312    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c6c435-0d51-44d3-a957-529175c479a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6c6c435-0d51-44d3-a957-529175c479a1" (UID: "c6c6c435-0d51-44d3-a957-529175c479a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:33.368510 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.368481    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6c6c435-0d51-44d3-a957-529175c479a1" (UID: "c6c6c435-0d51-44d3-a957-529175c479a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:33.368635 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.368521    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6c6c435-0d51-44d3-a957-529175c479a1" (UID: "c6c6c435-0d51-44d3-a957-529175c479a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:33.370249 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.370221    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c6c435-0d51-44d3-a957-529175c479a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6c6c435-0d51-44d3-a957-529175c479a1" (UID: "c6c6c435-0d51-44d3-a957-529175c479a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:33.370330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.370258    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c6c435-0d51-44d3-a957-529175c479a1-kube-api-access-jd2km" (OuterVolumeSpecName: "kube-api-access-jd2km") pod "c6c6c435-0d51-44d3-a957-529175c479a1" (UID: "c6c6c435-0d51-44d3-a957-529175c479a1"). InnerVolumeSpecName "kube-api-access-jd2km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:33.469088 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.469047    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c6c435-0d51-44d3-a957-529175c479a1-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:33.469088 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.469081    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c6c435-0d51-44d3-a957-529175c479a1-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:33.469088 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.469091    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:33.469303 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.469100    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2km\" (UniqueName: \"kubernetes.io/projected/c6c6c435-0d51-44d3-a957-529175c479a1-kube-api-access-jd2km\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:33.469303 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.469109    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c6c435-0d51-44d3-a957-529175c479a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:19:33.557354 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.557322    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6c6c435-0d51-44d3-a957-529175c479a1" containerID="be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab" exitCode=0
Sep 17 19:19:33.557511 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.557385    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"
Sep 17 19:19:33.557511 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.557408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerDied","Data":"be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab"}
Sep 17 19:19:33.557511 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.557474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn" event={"ID":"c6c6c435-0d51-44d3-a957-529175c479a1","Type":"ContainerDied","Data":"267e832508960e2dc3adc3dd6af1d97772002c31264bbb2e8dda9de707032bea"}
Sep 17 19:19:33.557511 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.557504    2583 scope.go:117] "RemoveContainer" containerID="be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab"
Sep 17 19:19:33.566223 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.566205    2583 scope.go:117] "RemoveContainer" containerID="e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08"
Sep 17 19:19:33.578796 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.578772    2583 scope.go:117] "RemoveContainer" containerID="3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134"
Sep 17 19:19:33.580835 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.580811    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"]
Sep 17 19:19:33.585488 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.585463    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-lxkbn"]
Sep 17 19:19:33.587391 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.587376    2583 scope.go:117] "RemoveContainer" containerID="772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0"
Sep 17 19:19:33.594853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.594840    2583 scope.go:117] "RemoveContainer" containerID="be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab"
Sep 17 19:19:33.595154 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:33.595138    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab\": container with ID starting with be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab not found: ID does not exist" containerID="be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab"
Sep 17 19:19:33.595205 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.595164    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab"} err="failed to get container status \"be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab\": rpc error: code = NotFound desc = could not find container \"be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab\": container with ID starting with be60963bd92c8d6e62c63139673c490f8e26001b53d3ff700fe59fde756a64ab not found: ID does not exist"
Sep 17 19:19:33.595205 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.595183    2583 scope.go:117] "RemoveContainer" containerID="e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08"
Sep 17 19:19:33.595432 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:33.595413    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08\": container with ID starting with e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08 not found: ID does not exist" containerID="e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08"
Sep 17 19:19:33.595475 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.595437    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08"} err="failed to get container status \"e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08\": rpc error: code = NotFound desc = could not find container \"e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08\": container with ID starting with e23379504fe73558e858223c611482967f503c348fdc259e336d653e2c87be08 not found: ID does not exist"
Sep 17 19:19:33.595475 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.595453    2583 scope.go:117] "RemoveContainer" containerID="3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134"
Sep 17 19:19:33.595687 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:33.595670    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134\": container with ID starting with 3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134 not found: ID does not exist" containerID="3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134"
Sep 17 19:19:33.595748 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.595694    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134"} err="failed to get container status \"3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134\": rpc error: code = NotFound desc = could not find container \"3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134\": container with ID starting with 3ae51ed0b6223aa4cca24a2bcdbe6cd0d8426011316a2b61e7595a800aa36134 not found: ID does not exist"
Sep 17 19:19:33.595748 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.595715    2583 scope.go:117] "RemoveContainer" containerID="772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0"
Sep 17 19:19:33.595935 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:19:33.595918    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0\": container with ID starting with 772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0 not found: ID does not exist" containerID="772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0"
Sep 17 19:19:33.595989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:33.595942    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0"} err="failed to get container status \"772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0\": rpc error: code = NotFound desc = could not find container \"772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0\": container with ID starting with 772a7d8c902396aa6ad123b68a1dcd53b7e29f3b36ef53052b4e4e6427a6b8b0 not found: ID does not exist"
Sep 17 19:19:35.038967 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:35.038935    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" path="/var/lib/kubelet/pods/c6c6c435-0d51-44d3-a957-529175c479a1/volumes"
Sep 17 19:19:36.548065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:36.548033    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:19:36.548698 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:36.548668    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:36.548974 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:36.548933    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:42.525255 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:42.525213    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:19:46.548692 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:46.548647    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:46.549327 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:46.549299    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:19:52.525579 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:52.525542    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:19:56.548659 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:56.548553    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:19:56.549122 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:19:56.549025    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:02.524642 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:02.524590    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:20:06.549327 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:06.549282    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:20:06.549898 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:06.549867    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:12.524782 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:12.524742    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:20:16.548571 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:16.548525    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:20:16.549086 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:16.549056    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:22.525545 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:22.525505    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:20:26.549265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:26.549216    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:20:26.549786 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:26.549761    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:32.524598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:32.524555    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:20:36.549448 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:36.549410    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:20:36.550003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:36.549981    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:40.034215 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:40.034172    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:20:40.034755 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:40.034529    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:20:42.525803 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:42.525776    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:20:48.982786 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:48.982755    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"]
Sep 17 19:20:48.983221 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:48.982983    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" containerID="cri-o://9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec" gracePeriod=30
Sep 17 19:20:48.983221 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:48.983029    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kube-rbac-proxy" containerID="cri-o://1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec" gracePeriod=30
Sep 17 19:20:49.919437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:49.919406    2583 generic.go:356] "Generic (PLEG): container finished" podID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerID="1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec" exitCode=2
Sep 17 19:20:49.919634 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:49.919473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" event={"ID":"8bafcb2b-52ad-4671-9d62-7a62199d9e8b","Type":"ContainerDied","Data":"1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec"}
Sep 17 19:20:50.034828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:50.034783    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:20:50.035187 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:50.035052    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:20:52.520487 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:52.520442    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 19:20:52.524957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:52.524934    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 19:20:54.166940 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.166902    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"]
Sep 17 19:20:54.167301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167257    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent"
Sep 17 19:20:54.167301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167270    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent"
Sep 17 19:20:54.167301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167282    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container"
Sep 17 19:20:54.167301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167288    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container"
Sep 17 19:20:54.167301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167298    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy"
Sep 17 19:20:54.167301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167304    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy"
Sep 17 19:20:54.167489 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167314    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="storage-initializer"
Sep 17 19:20:54.167489 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167320    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="storage-initializer"
Sep 17 19:20:54.167489 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167372    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kube-rbac-proxy"
Sep 17 19:20:54.167489 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167380    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="agent"
Sep 17 19:20:54.167489 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.167387    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c6c435-0d51-44d3-a957-529175c479a1" containerName="kserve-container"
Sep 17 19:20:54.170561 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.170546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.172782 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.172762    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 19:20:54.172888 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.172820    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 19:20:54.180427 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.180408    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"]
Sep 17 19:20:54.242141 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.242110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.242257 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.242157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45702a16-77b0-4fec-b3fa-ae67bae29592-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.242257 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.242231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.242342 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.242290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45702a16-77b0-4fec-b3fa-ae67bae29592-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.242342 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.242315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp4qs\" (UniqueName: \"kubernetes.io/projected/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-api-access-kp4qs\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.343683 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.343650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp4qs\" (UniqueName: \"kubernetes.io/projected/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-api-access-kp4qs\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.343853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.343702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.343853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.343734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45702a16-77b0-4fec-b3fa-ae67bae29592-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.343853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.343787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.343990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.343862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45702a16-77b0-4fec-b3fa-ae67bae29592-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.343990 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:20:54.343892    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 19:20:54.343990 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:20:54.343969    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45702a16-77b0-4fec-b3fa-ae67bae29592-proxy-tls podName:45702a16-77b0-4fec-b3fa-ae67bae29592 nodeName:}" failed. No retries permitted until 2026-09-17 19:20:54.843945942 +0000 UTC m=+873.971687334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45702a16-77b0-4fec-b3fa-ae67bae29592-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" (UID: "45702a16-77b0-4fec-b3fa-ae67bae29592") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 19:20:54.344311 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.344291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45702a16-77b0-4fec-b3fa-ae67bae29592-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.344459 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.344443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.344525 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.344509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.352373 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.352345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp4qs\" (UniqueName: \"kubernetes.io/projected/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-api-access-kp4qs\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.849673 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.849641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45702a16-77b0-4fec-b3fa-ae67bae29592-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.852600 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.852543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45702a16-77b0-4fec-b3fa-ae67bae29592-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:54.925919 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.925896    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:20:54.943457 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.943432    2583 generic.go:356] "Generic (PLEG): container finished" podID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerID="9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec" exitCode=0
Sep 17 19:20:54.943573 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.943494    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"
Sep 17 19:20:54.943573 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.943519    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" event={"ID":"8bafcb2b-52ad-4671-9d62-7a62199d9e8b","Type":"ContainerDied","Data":"9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec"}
Sep 17 19:20:54.943673 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.943592    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds" event={"ID":"8bafcb2b-52ad-4671-9d62-7a62199d9e8b","Type":"ContainerDied","Data":"be955b19b01cc72c194206d6ee850874fb478b3b57f7a4d2250e3f44bb30509c"}
Sep 17 19:20:54.943673 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.943646    2583 scope.go:117] "RemoveContainer" containerID="1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec"
Sep 17 19:20:54.950073 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.950052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kserve-provision-location\") pod \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") "
Sep 17 19:20:54.950197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.950084    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-proxy-tls\") pod \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") "
Sep 17 19:20:54.950197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.950103    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-cabundle-cert\") pod \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") "
Sep 17 19:20:54.950197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.950161    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-rbac-proxy-sar-config\") pod \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") "
Sep 17 19:20:54.950325 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.950257    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22n6g\" (UniqueName: \"kubernetes.io/projected/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-api-access-22n6g\") pod \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\" (UID: \"8bafcb2b-52ad-4671-9d62-7a62199d9e8b\") "
Sep 17 19:20:54.951285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.950395    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bafcb2b-52ad-4671-9d62-7a62199d9e8b" (UID: "8bafcb2b-52ad-4671-9d62-7a62199d9e8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:20:54.951285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.951141    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bafcb2b-52ad-4671-9d62-7a62199d9e8b" (UID: "8bafcb2b-52ad-4671-9d62-7a62199d9e8b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:54.951285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.951138    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bafcb2b-52ad-4671-9d62-7a62199d9e8b" (UID: "8bafcb2b-52ad-4671-9d62-7a62199d9e8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:54.955674 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.953660    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-api-access-22n6g" (OuterVolumeSpecName: "kube-api-access-22n6g") pod "8bafcb2b-52ad-4671-9d62-7a62199d9e8b" (UID: "8bafcb2b-52ad-4671-9d62-7a62199d9e8b"). InnerVolumeSpecName "kube-api-access-22n6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:20:54.955674 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.954191    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bafcb2b-52ad-4671-9d62-7a62199d9e8b" (UID: "8bafcb2b-52ad-4671-9d62-7a62199d9e8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:20:54.963522 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.963504    2583 scope.go:117] "RemoveContainer" containerID="9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec"
Sep 17 19:20:54.970812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.970792    2583 scope.go:117] "RemoveContainer" containerID="4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4"
Sep 17 19:20:54.978134 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.978116    2583 scope.go:117] "RemoveContainer" containerID="1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec"
Sep 17 19:20:54.978384 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:20:54.978365    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec\": container with ID starting with 1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec not found: ID does not exist" containerID="1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec"
Sep 17 19:20:54.978423 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.978391    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec"} err="failed to get container status \"1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec\": rpc error: code = NotFound desc = could not find container \"1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec\": container with ID starting with 1043d3a611e37fb7c67655ca077d6204e428c5d9dad5e3e5a1c26e4b67bdb8ec not found: ID does not exist"
Sep 17 19:20:54.978423 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.978408    2583 scope.go:117] "RemoveContainer" containerID="9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec"
Sep 17 19:20:54.978681 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:20:54.978662    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec\": container with ID starting with 9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec not found: ID does not exist" containerID="9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec"
Sep 17 19:20:54.978734 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.978686    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec"} err="failed to get container status \"9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec\": rpc error: code = NotFound desc = could not find container \"9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec\": container with ID starting with 9026f0f7f8d813c190d70d05302a237d15ce2cb943a2a9b132de6845bbde2fec not found: ID does not exist"
Sep 17 19:20:54.978734 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.978702    2583 scope.go:117] "RemoveContainer" containerID="4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4"
Sep 17 19:20:54.978928 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:20:54.978913    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4\": container with ID starting with 4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4 not found: ID does not exist" containerID="4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4"
Sep 17 19:20:54.978971 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:54.978933    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4"} err="failed to get container status \"4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4\": rpc error: code = NotFound desc = could not find container \"4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4\": container with ID starting with 4eea12f253b110c020079686a417047b90358b5290be11a16d2d0394defb91d4 not found: ID does not exist"
Sep 17 19:20:55.051515 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.051496    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22n6g\" (UniqueName: \"kubernetes.io/projected/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-api-access-22n6g\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:20:55.051594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.051515    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:20:55.051594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.051527    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:20:55.051594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.051536    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:20:55.051594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.051544    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bafcb2b-52ad-4671-9d62-7a62199d9e8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:20:55.080978 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.080949    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:20:55.260372 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.260348    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"]
Sep 17 19:20:55.261962 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.261941    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-5vsds"]
Sep 17 19:20:55.355812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.355788    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"]
Sep 17 19:20:55.358185 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:20:55.358159    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45702a16_77b0_4fec_b3fa_ae67bae29592.slice/crio-6ab45cd00f072b92c6cff858d00fffeb5630f89c39af6d7935517465e1444ae4 WatchSource:0}: Error finding container 6ab45cd00f072b92c6cff858d00fffeb5630f89c39af6d7935517465e1444ae4: Status 404 returned error can't find the container with id 6ab45cd00f072b92c6cff858d00fffeb5630f89c39af6d7935517465e1444ae4
Sep 17 19:20:55.951180 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.951143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" event={"ID":"45702a16-77b0-4fec-b3fa-ae67bae29592","Type":"ContainerStarted","Data":"11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a"}
Sep 17 19:20:55.951360 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:55.951209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" event={"ID":"45702a16-77b0-4fec-b3fa-ae67bae29592","Type":"ContainerStarted","Data":"6ab45cd00f072b92c6cff858d00fffeb5630f89c39af6d7935517465e1444ae4"}
Sep 17 19:20:57.038599 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:57.038564    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" path="/var/lib/kubelet/pods/8bafcb2b-52ad-4671-9d62-7a62199d9e8b/volumes"
Sep 17 19:20:59.969333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:59.969304    2583 generic.go:356] "Generic (PLEG): container finished" podID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerID="11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a" exitCode=0
Sep 17 19:20:59.969716 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:20:59.969386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" event={"ID":"45702a16-77b0-4fec-b3fa-ae67bae29592","Type":"ContainerDied","Data":"11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a"}
Sep 17 19:21:00.975508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:00.975476    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" event={"ID":"45702a16-77b0-4fec-b3fa-ae67bae29592","Type":"ContainerStarted","Data":"fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8"}
Sep 17 19:21:00.975883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:00.975557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" event={"ID":"45702a16-77b0-4fec-b3fa-ae67bae29592","Type":"ContainerStarted","Data":"36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7"}
Sep 17 19:21:00.975883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:00.975769    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:21:00.998743 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:00.998696    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podStartSLOduration=3.20934351 podStartE2EDuration="6.998681036s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.789337526s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:00.996786637 +0000 UTC m=+880.124528033" watchObservedRunningTime="2026-09-17 19:21:00.998681036 +0000 UTC m=+880.126422437"
Sep 17 19:21:01.980127 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:01.980101    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:21:02.703831 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.703800    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-pq2fl_3cac9a39-7017-4bd5-b81e-7159ea25710d/kserve-container/0.log"
Sep 17 19:21:02.871208 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.871177    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"]
Sep 17 19:21:02.871512 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.871472    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" containerID="cri-o://eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17" gracePeriod=30
Sep 17 19:21:02.871687 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.871498    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" containerID="cri-o://6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e" gracePeriod=30
Sep 17 19:21:02.871687 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.871524    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy" containerID="cri-o://48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222" gracePeriod=30
Sep 17 19:21:02.889997 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.889967    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"]
Sep 17 19:21:02.890257 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.890225    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl" podUID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerName="kserve-container" containerID="cri-o://84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248" gracePeriod=30
Sep 17 19:21:02.890374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.890313    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl" podUID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerName="kube-rbac-proxy" containerID="cri-o://8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546" gracePeriod=30
Sep 17 19:21:02.954738 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.954665    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"]
Sep 17 19:21:02.955031 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.955019    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container"
Sep 17 19:21:02.955069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.955032    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container"
Sep 17 19:21:02.955069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.955043    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="storage-initializer"
Sep 17 19:21:02.955069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.955048    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="storage-initializer"
Sep 17 19:21:02.955069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.955057    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kube-rbac-proxy"
Sep 17 19:21:02.955069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.955062    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kube-rbac-proxy"
Sep 17 19:21:02.955218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.955127    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kserve-container"
Sep 17 19:21:02.955218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.955138    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bafcb2b-52ad-4671-9d62-7a62199d9e8b" containerName="kube-rbac-proxy"
Sep 17 19:21:02.957288 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.957271    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:02.959660 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.959636    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 19:21:02.959792 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.959774    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 19:21:02.968952 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:02.968931    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"]
Sep 17 19:21:03.019413 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.019370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a446110-2d66-4685-90b4-7ae80a30ba54-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.019744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.019436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.019744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.019479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.019744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.019579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh5zv\" (UniqueName: \"kubernetes.io/projected/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-api-access-xh5zv\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.019912 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.019758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a446110-2d66-4685-90b4-7ae80a30ba54-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.121154 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.121128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a446110-2d66-4685-90b4-7ae80a30ba54-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.121258 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.121172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.121258 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.121195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.121391 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.121368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh5zv\" (UniqueName: \"kubernetes.io/projected/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-api-access-xh5zv\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.121455 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.121437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a446110-2d66-4685-90b4-7ae80a30ba54-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.121562 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.121539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a446110-2d66-4685-90b4-7ae80a30ba54-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.121950 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.121931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.122366 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.122343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.123930 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.123908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a446110-2d66-4685-90b4-7ae80a30ba54-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.135887 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.135870    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:21:03.136778 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.136762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh5zv\" (UniqueName: \"kubernetes.io/projected/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-api-access-xh5zv\") pod \"isvc-autogluon-v1-predictor-574b5b7495-njj57\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.222449 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.222373    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cac9a39-7017-4bd5-b81e-7159ea25710d-proxy-tls\") pod \"3cac9a39-7017-4bd5-b81e-7159ea25710d\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") "
Sep 17 19:21:03.222449 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.222429    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-rbac-proxy-sar-config\") pod \"3cac9a39-7017-4bd5-b81e-7159ea25710d\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") "
Sep 17 19:21:03.222643 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.222453    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nckcg\" (UniqueName: \"kubernetes.io/projected/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-api-access-nckcg\") pod \"3cac9a39-7017-4bd5-b81e-7159ea25710d\" (UID: \"3cac9a39-7017-4bd5-b81e-7159ea25710d\") "
Sep 17 19:21:03.222805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.222784    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3cac9a39-7017-4bd5-b81e-7159ea25710d" (UID: "3cac9a39-7017-4bd5-b81e-7159ea25710d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:21:03.224632 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.224598    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-api-access-nckcg" (OuterVolumeSpecName: "kube-api-access-nckcg") pod "3cac9a39-7017-4bd5-b81e-7159ea25710d" (UID: "3cac9a39-7017-4bd5-b81e-7159ea25710d"). InnerVolumeSpecName "kube-api-access-nckcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:21:03.224702 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.224652    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cac9a39-7017-4bd5-b81e-7159ea25710d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cac9a39-7017-4bd5-b81e-7159ea25710d" (UID: "3cac9a39-7017-4bd5-b81e-7159ea25710d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:21:03.283773 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.283740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:21:03.324006 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.323971    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cac9a39-7017-4bd5-b81e-7159ea25710d-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:21:03.324006 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.324009    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:21:03.324204 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.324027    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nckcg\" (UniqueName: \"kubernetes.io/projected/3cac9a39-7017-4bd5-b81e-7159ea25710d-kube-api-access-nckcg\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:21:03.557373 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.557349    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"]
Sep 17 19:21:03.559361 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:21:03.559335    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a446110_2d66_4685_90b4_7ae80a30ba54.slice/crio-869c7bda5fdb00f06f04d12accbb397c89d33011c43137c4b4e62abefbb7c64e WatchSource:0}: Error finding container 869c7bda5fdb00f06f04d12accbb397c89d33011c43137c4b4e62abefbb7c64e: Status 404 returned error can't find the container with id 869c7bda5fdb00f06f04d12accbb397c89d33011c43137c4b4e62abefbb7c64e
Sep 17 19:21:03.991642 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.991589    2583 generic.go:356] "Generic (PLEG): container finished" podID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerID="8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546" exitCode=2
Sep 17 19:21:03.991642 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.991628    2583 generic.go:356] "Generic (PLEG): container finished" podID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerID="84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248" exitCode=2
Sep 17 19:21:03.991875 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.991675    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"
Sep 17 19:21:03.991875 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.991674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl" event={"ID":"3cac9a39-7017-4bd5-b81e-7159ea25710d","Type":"ContainerDied","Data":"8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546"}
Sep 17 19:21:03.991875 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.991810    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl" event={"ID":"3cac9a39-7017-4bd5-b81e-7159ea25710d","Type":"ContainerDied","Data":"84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248"}
Sep 17 19:21:03.991875 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.991848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl" event={"ID":"3cac9a39-7017-4bd5-b81e-7159ea25710d","Type":"ContainerDied","Data":"a1f0605451dc578a60b38d06efb3b3bc99ff916da656885a7cefb40ae2420c65"}
Sep 17 19:21:03.992056 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.991883    2583 scope.go:117] "RemoveContainer" containerID="8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546"
Sep 17 19:21:03.994573 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.994549    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerID="48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222" exitCode=2
Sep 17 19:21:03.994697 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.994628    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerDied","Data":"48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222"}
Sep 17 19:21:03.996446 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.996422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" event={"ID":"4a446110-2d66-4685-90b4-7ae80a30ba54","Type":"ContainerStarted","Data":"cef7a564f529e29d50dbb1b64180030518b390222424a6b285bc1ba0f776ad41"}
Sep 17 19:21:03.996564 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:03.996479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" event={"ID":"4a446110-2d66-4685-90b4-7ae80a30ba54","Type":"ContainerStarted","Data":"869c7bda5fdb00f06f04d12accbb397c89d33011c43137c4b4e62abefbb7c64e"}
Sep 17 19:21:04.001205 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.001189    2583 scope.go:117] "RemoveContainer" containerID="84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248"
Sep 17 19:21:04.010427 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.010412    2583 scope.go:117] "RemoveContainer" containerID="8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546"
Sep 17 19:21:04.010730 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:21:04.010709    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546\": container with ID starting with 8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546 not found: ID does not exist" containerID="8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546"
Sep 17 19:21:04.010788 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.010737    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546"} err="failed to get container status \"8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546\": rpc error: code = NotFound desc = could not find container \"8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546\": container with ID starting with 8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546 not found: ID does not exist"
Sep 17 19:21:04.010788 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.010751    2583 scope.go:117] "RemoveContainer" containerID="84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248"
Sep 17 19:21:04.011041 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:21:04.011024    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248\": container with ID starting with 84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248 not found: ID does not exist" containerID="84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248"
Sep 17 19:21:04.011096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.011046    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248"} err="failed to get container status \"84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248\": rpc error: code = NotFound desc = could not find container \"84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248\": container with ID starting with 84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248 not found: ID does not exist"
Sep 17 19:21:04.011096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.011063    2583 scope.go:117] "RemoveContainer" containerID="8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546"
Sep 17 19:21:04.011297 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.011281    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546"} err="failed to get container status \"8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546\": rpc error: code = NotFound desc = could not find container \"8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546\": container with ID starting with 8b2c611f03a31821ccf4e0f590ea4d56a03e6b732df226d41261cc82f9821546 not found: ID does not exist"
Sep 17 19:21:04.011332 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.011299    2583 scope.go:117] "RemoveContainer" containerID="84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248"
Sep 17 19:21:04.011565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.011549    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248"} err="failed to get container status \"84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248\": rpc error: code = NotFound desc = could not find container \"84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248\": container with ID starting with 84593652a0f61cc22320cccc75d3a033ae46798359ef1ba086906c0d98965248 not found: ID does not exist"
Sep 17 19:21:04.032065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.032043    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"]
Sep 17 19:21:04.036488 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:04.036469    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pq2fl"]
Sep 17 19:21:05.045485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:05.045442    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cac9a39-7017-4bd5-b81e-7159ea25710d" path="/var/lib/kubelet/pods/3cac9a39-7017-4bd5-b81e-7159ea25710d/volumes"
Sep 17 19:21:06.544119 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:06.544080    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 19:21:07.989212 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:07.989191    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:21:08.020877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:08.020854    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerID="eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17" exitCode=0
Sep 17 19:21:08.020996 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:08.020917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerDied","Data":"eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17"}
Sep 17 19:21:10.034901 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:10.034855    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:21:10.035276 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:10.035176    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:11.544758 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:11.544720    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 19:21:16.544299 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:16.544257    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 19:21:16.544749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:16.544382    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:21:20.035098 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:20.035043    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:21:20.035497 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:20.035412    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:21.545208 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:21.545162    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 19:21:22.966237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:22.966209    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:21:22.966969 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:22.966954    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:21:24.103914 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:24.103814    2583 generic.go:356] "Generic (PLEG): container finished" podID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerID="cef7a564f529e29d50dbb1b64180030518b390222424a6b285bc1ba0f776ad41" exitCode=0
Sep 17 19:21:24.103914 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:24.103852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" event={"ID":"4a446110-2d66-4685-90b4-7ae80a30ba54","Type":"ContainerDied","Data":"cef7a564f529e29d50dbb1b64180030518b390222424a6b285bc1ba0f776ad41"}
Sep 17 19:21:26.545225 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:26.545178    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 19:21:30.034693 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:30.034629    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 19:21:30.037367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:30.034777    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:21:30.037367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:30.035213    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:21:30.037367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:30.035363    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:21:31.545177 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:31.545129    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 19:21:33.031106 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.031082    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:21:33.154302 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.154214    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerID="6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e" exitCode=0
Sep 17 19:21:33.154302 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.154284    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"
Sep 17 19:21:33.154302 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.154287    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerDied","Data":"6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e"}
Sep 17 19:21:33.154550 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.154361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6" event={"ID":"4e50d729-a900-4852-b1c4-594aea5acd8e","Type":"ContainerDied","Data":"9818a9f3f081db4881b7fe7c81a86c360ead5b699f1c136b340a617b6a6987ca"}
Sep 17 19:21:33.154550 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.154405    2583 scope.go:117] "RemoveContainer" containerID="6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e"
Sep 17 19:21:33.163812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.163785    2583 scope.go:117] "RemoveContainer" containerID="48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222"
Sep 17 19:21:33.171694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.171674    2583 scope.go:117] "RemoveContainer" containerID="eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17"
Sep 17 19:21:33.179054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.179030    2583 scope.go:117] "RemoveContainer" containerID="1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3"
Sep 17 19:21:33.186196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.186175    2583 scope.go:117] "RemoveContainer" containerID="6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e"
Sep 17 19:21:33.186476 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:21:33.186458    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e\": container with ID starting with 6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e not found: ID does not exist" containerID="6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e"
Sep 17 19:21:33.186524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.186484    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e"} err="failed to get container status \"6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e\": rpc error: code = NotFound desc = could not find container \"6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e\": container with ID starting with 6ad286a8fc5e0e9a631543868fcdf8829675f85dbcaa93173deabe34611d558e not found: ID does not exist"
Sep 17 19:21:33.186524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.186501    2583 scope.go:117] "RemoveContainer" containerID="48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222"
Sep 17 19:21:33.186803 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:21:33.186785    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222\": container with ID starting with 48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222 not found: ID does not exist" containerID="48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222"
Sep 17 19:21:33.186865 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.186809    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222"} err="failed to get container status \"48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222\": rpc error: code = NotFound desc = could not find container \"48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222\": container with ID starting with 48a6666752a3e4b09941ab71c595fff88b1b725d1f9f148b4308638d4130f222 not found: ID does not exist"
Sep 17 19:21:33.186865 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.186824    2583 scope.go:117] "RemoveContainer" containerID="eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17"
Sep 17 19:21:33.187105 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:21:33.187088    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17\": container with ID starting with eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17 not found: ID does not exist" containerID="eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17"
Sep 17 19:21:33.187158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.187107    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17"} err="failed to get container status \"eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17\": rpc error: code = NotFound desc = could not find container \"eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17\": container with ID starting with eb5d44a276bad90ca1517a5c25862a3fc72b124c982a8cc3717ba11c31c99c17 not found: ID does not exist"
Sep 17 19:21:33.187158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.187119    2583 scope.go:117] "RemoveContainer" containerID="1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3"
Sep 17 19:21:33.187368 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:21:33.187353    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3\": container with ID starting with 1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3 not found: ID does not exist" containerID="1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3"
Sep 17 19:21:33.187418 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.187370    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3"} err="failed to get container status \"1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3\": rpc error: code = NotFound desc = could not find container \"1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3\": container with ID starting with 1f69d68fc7753a9de1be0a07f49e576363010e3401290d53426ebe38059693d3 not found: ID does not exist"
Sep 17 19:21:33.191744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.191727    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts5bs\" (UniqueName: \"kubernetes.io/projected/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-api-access-ts5bs\") pod \"4e50d729-a900-4852-b1c4-594aea5acd8e\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") "
Sep 17 19:21:33.191810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.191761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-cabundle-cert\") pod \"4e50d729-a900-4852-b1c4-594aea5acd8e\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") "
Sep 17 19:21:33.191810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.191782    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e50d729-a900-4852-b1c4-594aea5acd8e-kserve-provision-location\") pod \"4e50d729-a900-4852-b1c4-594aea5acd8e\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") "
Sep 17 19:21:33.191810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.191801    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-rbac-proxy-sar-config\") pod \"4e50d729-a900-4852-b1c4-594aea5acd8e\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") "
Sep 17 19:21:33.191952 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.191820    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e50d729-a900-4852-b1c4-594aea5acd8e-proxy-tls\") pod \"4e50d729-a900-4852-b1c4-594aea5acd8e\" (UID: \"4e50d729-a900-4852-b1c4-594aea5acd8e\") "
Sep 17 19:21:33.192121 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.192085    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e50d729-a900-4852-b1c4-594aea5acd8e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e50d729-a900-4852-b1c4-594aea5acd8e" (UID: "4e50d729-a900-4852-b1c4-594aea5acd8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:33.192193 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.192146    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e50d729-a900-4852-b1c4-594aea5acd8e" (UID: "4e50d729-a900-4852-b1c4-594aea5acd8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:21:33.192193 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.192157    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e50d729-a900-4852-b1c4-594aea5acd8e" (UID: "4e50d729-a900-4852-b1c4-594aea5acd8e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:21:33.194375 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.194344    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-api-access-ts5bs" (OuterVolumeSpecName: "kube-api-access-ts5bs") pod "4e50d729-a900-4852-b1c4-594aea5acd8e" (UID: "4e50d729-a900-4852-b1c4-594aea5acd8e"). InnerVolumeSpecName "kube-api-access-ts5bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:21:33.194471 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.194424    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e50d729-a900-4852-b1c4-594aea5acd8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e50d729-a900-4852-b1c4-594aea5acd8e" (UID: "4e50d729-a900-4852-b1c4-594aea5acd8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:21:33.293168 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.293135    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5bs\" (UniqueName: \"kubernetes.io/projected/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-api-access-ts5bs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:21:33.293168 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.293161    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:21:33.293168 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.293171    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e50d729-a900-4852-b1c4-594aea5acd8e-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:21:33.293168 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.293179    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e50d729-a900-4852-b1c4-594aea5acd8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:21:33.293421 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.293188    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e50d729-a900-4852-b1c4-594aea5acd8e-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:21:33.478752 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.478726    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"]
Sep 17 19:21:33.482248 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:33.482226    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-cjhd6"]
Sep 17 19:21:35.038828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:35.038794    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" path="/var/lib/kubelet/pods/4e50d729-a900-4852-b1c4-594aea5acd8e/volumes"
Sep 17 19:21:37.990209 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:37.990173    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:21:47.990117 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:47.990067    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:21:57.990232 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:21:57.990191    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:22:07.989987 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:07.989938    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:22:17.990055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:17.990005    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:22:28.646085 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:28.646048    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:22:34.269051 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:34.268959    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"]
Sep 17 19:22:34.269472 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:34.269241    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" containerID="cri-o://36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7" gracePeriod=30
Sep 17 19:22:34.269472 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:34.269292    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" containerID="cri-o://fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8" gracePeriod=30
Sep 17 19:22:34.450339 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:34.450301    2583 generic.go:356] "Generic (PLEG): container finished" podID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerID="fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8" exitCode=2
Sep 17 19:22:34.450491 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:34.450380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" event={"ID":"45702a16-77b0-4fec-b3fa-ae67bae29592","Type":"ContainerDied","Data":"fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8"}
Sep 17 19:22:37.985763 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:37.985717    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:22:37.990272 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:37.990236    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:22:39.444900 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.444849    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"]
Sep 17 19:22:39.445377 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445358    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent"
Sep 17 19:22:39.445453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445382    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent"
Sep 17 19:22:39.445453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445398    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy"
Sep 17 19:22:39.445453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445407    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy"
Sep 17 19:22:39.445453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445420    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerName="kube-rbac-proxy"
Sep 17 19:22:39.445453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445429    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerName="kube-rbac-proxy"
Sep 17 19:22:39.445453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445442    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="storage-initializer"
Sep 17 19:22:39.445453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445450    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="storage-initializer"
Sep 17 19:22:39.445824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445468    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerName="kserve-container"
Sep 17 19:22:39.445824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445476    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerName="kserve-container"
Sep 17 19:22:39.445824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445493    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container"
Sep 17 19:22:39.445824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445501    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container"
Sep 17 19:22:39.445824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445586    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="agent"
Sep 17 19:22:39.445824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445599    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerName="kube-rbac-proxy"
Sep 17 19:22:39.445824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445608    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cac9a39-7017-4bd5-b81e-7159ea25710d" containerName="kserve-container"
Sep 17 19:22:39.445824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445637    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kserve-container"
Sep 17 19:22:39.445824 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.445647    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e50d729-a900-4852-b1c4-594aea5acd8e" containerName="kube-rbac-proxy"
Sep 17 19:22:39.471551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.471517    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"]
Sep 17 19:22:39.471728 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.471676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.474485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.474462    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 19:22:39.474957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.474928    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 19:22:39.596328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.596292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30590ad0-2511-4af8-bd1a-62a36d7221ee-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.596328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.596334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30590ad0-2511-4af8-bd1a-62a36d7221ee-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.596559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.596381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.596559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.596519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxnj\" (UniqueName: \"kubernetes.io/projected/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-api-access-vxxnj\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.596559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.596548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.697276 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.697183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxnj\" (UniqueName: \"kubernetes.io/projected/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-api-access-vxxnj\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.697276 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.697221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.697276 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.697268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30590ad0-2511-4af8-bd1a-62a36d7221ee-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.697560 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.697294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30590ad0-2511-4af8-bd1a-62a36d7221ee-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.697560 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.697352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.697560 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:22:39.697529    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 17 19:22:39.697746 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:22:39.697638    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30590ad0-2511-4af8-bd1a-62a36d7221ee-proxy-tls podName:30590ad0-2511-4af8-bd1a-62a36d7221ee nodeName:}" failed. No retries permitted until 2026-09-17 19:22:40.197594638 +0000 UTC m=+979.325336019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30590ad0-2511-4af8-bd1a-62a36d7221ee-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" (UID: "30590ad0-2511-4af8-bd1a-62a36d7221ee") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 17 19:22:39.697936 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.697911    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30590ad0-2511-4af8-bd1a-62a36d7221ee-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.698147 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.698130    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.698196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.698137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:39.710918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:39.710882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxnj\" (UniqueName: \"kubernetes.io/projected/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-api-access-vxxnj\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:40.203305 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:40.203262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30590ad0-2511-4af8-bd1a-62a36d7221ee-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:40.206285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:40.206259    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30590ad0-2511-4af8-bd1a-62a36d7221ee-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:40.383578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:40.383536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:22:42.984828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:42.984781    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:22:47.985498 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:47.985445    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:22:47.986005 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:47.985596    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:22:47.989868 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:47.989817    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:22:52.985486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:52.985445    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:22:57.985414 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:57.985369    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:22:57.989944 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:57.989907    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:22:57.990056 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:22:57.990029    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:23:02.985637 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:02.985536    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:23:07.985554 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:07.985496    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:23:07.990422 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:07.990393    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:23:12.985212 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:12.985163    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:23:17.985714 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:17.985656    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:23:17.989987 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:17.989957    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 19:23:22.985309 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:22.985263    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 19:23:25.075093 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.075019    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:23:25.110070 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.110043    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-cabundle-cert\") pod \"45702a16-77b0-4fec-b3fa-ae67bae29592\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") "
Sep 17 19:23:25.110237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.110113    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45702a16-77b0-4fec-b3fa-ae67bae29592-kserve-provision-location\") pod \"45702a16-77b0-4fec-b3fa-ae67bae29592\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") "
Sep 17 19:23:25.110237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.110140    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-rbac-proxy-sar-config\") pod \"45702a16-77b0-4fec-b3fa-ae67bae29592\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") "
Sep 17 19:23:25.110237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.110211    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp4qs\" (UniqueName: \"kubernetes.io/projected/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-api-access-kp4qs\") pod \"45702a16-77b0-4fec-b3fa-ae67bae29592\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") "
Sep 17 19:23:25.110429 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.110241    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45702a16-77b0-4fec-b3fa-ae67bae29592-proxy-tls\") pod \"45702a16-77b0-4fec-b3fa-ae67bae29592\" (UID: \"45702a16-77b0-4fec-b3fa-ae67bae29592\") "
Sep 17 19:23:25.110505 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.110466    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45702a16-77b0-4fec-b3fa-ae67bae29592-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45702a16-77b0-4fec-b3fa-ae67bae29592" (UID: "45702a16-77b0-4fec-b3fa-ae67bae29592"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:25.110505 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.110491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45702a16-77b0-4fec-b3fa-ae67bae29592" (UID: "45702a16-77b0-4fec-b3fa-ae67bae29592"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:23:25.110833 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.110506    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45702a16-77b0-4fec-b3fa-ae67bae29592" (UID: "45702a16-77b0-4fec-b3fa-ae67bae29592"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:23:25.112463 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.112437    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-api-access-kp4qs" (OuterVolumeSpecName: "kube-api-access-kp4qs") pod "45702a16-77b0-4fec-b3fa-ae67bae29592" (UID: "45702a16-77b0-4fec-b3fa-ae67bae29592"). InnerVolumeSpecName "kube-api-access-kp4qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:25.112573 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.112555    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45702a16-77b0-4fec-b3fa-ae67bae29592-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45702a16-77b0-4fec-b3fa-ae67bae29592" (UID: "45702a16-77b0-4fec-b3fa-ae67bae29592"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:25.211218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.211158    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45702a16-77b0-4fec-b3fa-ae67bae29592-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:25.211218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.211192    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:25.211218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.211209    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp4qs\" (UniqueName: \"kubernetes.io/projected/45702a16-77b0-4fec-b3fa-ae67bae29592-kube-api-access-kp4qs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:25.211425 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.211226    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45702a16-77b0-4fec-b3fa-ae67bae29592-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:25.211425 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.211242    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45702a16-77b0-4fec-b3fa-ae67bae29592-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:25.433639 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.433577    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"]
Sep 17 19:23:25.435756 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:23:25.435734    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30590ad0_2511_4af8_bd1a_62a36d7221ee.slice/crio-24029e083e1c078086c19a02b3511e29475fa20dfae6ada90ffe54cd36c34cc8 WatchSource:0}: Error finding container 24029e083e1c078086c19a02b3511e29475fa20dfae6ada90ffe54cd36c34cc8: Status 404 returned error can't find the container with id 24029e083e1c078086c19a02b3511e29475fa20dfae6ada90ffe54cd36c34cc8
Sep 17 19:23:25.703951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.703857    2583 generic.go:356] "Generic (PLEG): container finished" podID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerID="36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7" exitCode=0
Sep 17 19:23:25.703951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.703911    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"
Sep 17 19:23:25.704280 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.704000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" event={"ID":"45702a16-77b0-4fec-b3fa-ae67bae29592","Type":"ContainerDied","Data":"36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7"}
Sep 17 19:23:25.704280 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.704073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd" event={"ID":"45702a16-77b0-4fec-b3fa-ae67bae29592","Type":"ContainerDied","Data":"6ab45cd00f072b92c6cff858d00fffeb5630f89c39af6d7935517465e1444ae4"}
Sep 17 19:23:25.704280 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.704111    2583 scope.go:117] "RemoveContainer" containerID="fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8"
Sep 17 19:23:25.705954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.705931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" event={"ID":"30590ad0-2511-4af8-bd1a-62a36d7221ee","Type":"ContainerStarted","Data":"712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f"}
Sep 17 19:23:25.706128 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.706105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" event={"ID":"30590ad0-2511-4af8-bd1a-62a36d7221ee","Type":"ContainerStarted","Data":"24029e083e1c078086c19a02b3511e29475fa20dfae6ada90ffe54cd36c34cc8"}
Sep 17 19:23:25.714910 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.714889    2583 scope.go:117] "RemoveContainer" containerID="36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7"
Sep 17 19:23:25.727338 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.727304    2583 scope.go:117] "RemoveContainer" containerID="11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a"
Sep 17 19:23:25.737308 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.737286    2583 scope.go:117] "RemoveContainer" containerID="fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8"
Sep 17 19:23:25.737608 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:23:25.737588    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8\": container with ID starting with fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8 not found: ID does not exist" containerID="fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8"
Sep 17 19:23:25.737696 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.737645    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8"} err="failed to get container status \"fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8\": rpc error: code = NotFound desc = could not find container \"fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8\": container with ID starting with fe7333d74b5515af5b1de8165ecf582d85641032b8f7e3892072f0e3e88d24e8 not found: ID does not exist"
Sep 17 19:23:25.737696 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.737665    2583 scope.go:117] "RemoveContainer" containerID="36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7"
Sep 17 19:23:25.738020 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:23:25.737979    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7\": container with ID starting with 36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7 not found: ID does not exist" containerID="36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7"
Sep 17 19:23:25.738111 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.738013    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7"} err="failed to get container status \"36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7\": rpc error: code = NotFound desc = could not find container \"36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7\": container with ID starting with 36140f9ba80b8f8a16466e07500e1d33c82855044849081ddc9ea26c949a5ce7 not found: ID does not exist"
Sep 17 19:23:25.738111 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.738042    2583 scope.go:117] "RemoveContainer" containerID="11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a"
Sep 17 19:23:25.738426 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:23:25.738398    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a\": container with ID starting with 11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a not found: ID does not exist" containerID="11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a"
Sep 17 19:23:25.738536 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.738433    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a"} err="failed to get container status \"11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a\": rpc error: code = NotFound desc = could not find container \"11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a\": container with ID starting with 11a147b19cac41981ac192b87233241a122c67ccfbe1afe90310ca3a53047e0a not found: ID does not exist"
Sep 17 19:23:25.740351 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.740332    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"]
Sep 17 19:23:25.743734 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:25.743713    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-npfmd"]
Sep 17 19:23:26.713431 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:26.713393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" event={"ID":"4a446110-2d66-4685-90b4-7ae80a30ba54","Type":"ContainerStarted","Data":"9d58894d9454c18d82a7514c15d30e08cbf1a87aa2b95750b2ddeb510bf7527c"}
Sep 17 19:23:26.713969 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:26.713468    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:23:26.713969 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:26.713503    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" event={"ID":"4a446110-2d66-4685-90b4-7ae80a30ba54","Type":"ContainerStarted","Data":"cb7454e1dd87f32a1dabec75a166081cd9bd8199f3bd1fe49cb9c9ec37658699"}
Sep 17 19:23:26.713969 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:26.713535    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:23:26.714814 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:26.714786    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:23:26.735880 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:26.735829    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" podStartSLOduration=2.756376344 podStartE2EDuration="2m24.73581138s" totalImagesPullingTime="2m1.883717825s" totalInitContainerRuntime="20.095717211s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:23:26.734571585 +0000 UTC m=+1025.862312986" watchObservedRunningTime="2026-09-17 19:23:26.73581138 +0000 UTC m=+1025.863552779"
Sep 17 19:23:27.039097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:27.039020    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" path="/var/lib/kubelet/pods/45702a16-77b0-4fec-b3fa-ae67bae29592/volumes"
Sep 17 19:23:27.719927 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:27.719884    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:23:29.729814 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:29.729782    2583 generic.go:356] "Generic (PLEG): container finished" podID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerID="712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f" exitCode=0
Sep 17 19:23:29.730187 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:29.729864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" event={"ID":"30590ad0-2511-4af8-bd1a-62a36d7221ee","Type":"ContainerDied","Data":"712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f"}
Sep 17 19:23:30.736370 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:30.736339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" event={"ID":"30590ad0-2511-4af8-bd1a-62a36d7221ee","Type":"ContainerStarted","Data":"3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5"}
Sep 17 19:23:30.736770 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:30.736414    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" event={"ID":"30590ad0-2511-4af8-bd1a-62a36d7221ee","Type":"ContainerStarted","Data":"439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478"}
Sep 17 19:23:30.736770 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:30.736629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:23:30.758290 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:30.758226    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" podStartSLOduration=48.060328121 podStartE2EDuration="51.758202331s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.69787421s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:23:30.756261936 +0000 UTC m=+1029.884003338" watchObservedRunningTime="2026-09-17 19:23:30.758202331 +0000 UTC m=+1029.885943740"
Sep 17 19:23:31.740056 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:31.740026    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:23:32.724731 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:32.724707    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:23:32.725211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:32.725180    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 19:23:37.750218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:37.750185    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:23:42.726667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:42.726637    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:23:49.391712 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:49.391677    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"]
Sep 17 19:23:49.392084 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:49.391917    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kserve-container" containerID="cri-o://cb7454e1dd87f32a1dabec75a166081cd9bd8199f3bd1fe49cb9c9ec37658699" gracePeriod=30
Sep 17 19:23:49.392084 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:49.392003    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kube-rbac-proxy" containerID="cri-o://9d58894d9454c18d82a7514c15d30e08cbf1a87aa2b95750b2ddeb510bf7527c" gracePeriod=30
Sep 17 19:23:49.822958 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:49.822924    2583 generic.go:356] "Generic (PLEG): container finished" podID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerID="9d58894d9454c18d82a7514c15d30e08cbf1a87aa2b95750b2ddeb510bf7527c" exitCode=2
Sep 17 19:23:49.823128 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:49.822998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" event={"ID":"4a446110-2d66-4685-90b4-7ae80a30ba54","Type":"ContainerDied","Data":"9d58894d9454c18d82a7514c15d30e08cbf1a87aa2b95750b2ddeb510bf7527c"}
Sep 17 19:23:50.829050 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:50.829014    2583 generic.go:356] "Generic (PLEG): container finished" podID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerID="cb7454e1dd87f32a1dabec75a166081cd9bd8199f3bd1fe49cb9c9ec37658699" exitCode=0
Sep 17 19:23:50.829513 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:50.829084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" event={"ID":"4a446110-2d66-4685-90b4-7ae80a30ba54","Type":"ContainerDied","Data":"cb7454e1dd87f32a1dabec75a166081cd9bd8199f3bd1fe49cb9c9ec37658699"}
Sep 17 19:23:50.829513 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:50.829138    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57" event={"ID":"4a446110-2d66-4685-90b4-7ae80a30ba54","Type":"ContainerDied","Data":"869c7bda5fdb00f06f04d12accbb397c89d33011c43137c4b4e62abefbb7c64e"}
Sep 17 19:23:50.829513 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:50.829160    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="869c7bda5fdb00f06f04d12accbb397c89d33011c43137c4b4e62abefbb7c64e"
Sep 17 19:23:50.860872 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:50.860809    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:23:51.044172 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.044143    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a446110-2d66-4685-90b4-7ae80a30ba54-proxy-tls\") pod \"4a446110-2d66-4685-90b4-7ae80a30ba54\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") "
Sep 17 19:23:51.044361 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.044186    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-rbac-proxy-sar-config\") pod \"4a446110-2d66-4685-90b4-7ae80a30ba54\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") "
Sep 17 19:23:51.044361 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.044225    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a446110-2d66-4685-90b4-7ae80a30ba54-kserve-provision-location\") pod \"4a446110-2d66-4685-90b4-7ae80a30ba54\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") "
Sep 17 19:23:51.044361 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.044244    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-cabundle-cert\") pod \"4a446110-2d66-4685-90b4-7ae80a30ba54\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") "
Sep 17 19:23:51.044361 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.044278    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh5zv\" (UniqueName: \"kubernetes.io/projected/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-api-access-xh5zv\") pod \"4a446110-2d66-4685-90b4-7ae80a30ba54\" (UID: \"4a446110-2d66-4685-90b4-7ae80a30ba54\") "
Sep 17 19:23:51.044657 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.044632    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a446110-2d66-4685-90b4-7ae80a30ba54" (UID: "4a446110-2d66-4685-90b4-7ae80a30ba54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:23:51.044727 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.044646    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a446110-2d66-4685-90b4-7ae80a30ba54" (UID: "4a446110-2d66-4685-90b4-7ae80a30ba54"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:23:51.046514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.046486    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a446110-2d66-4685-90b4-7ae80a30ba54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a446110-2d66-4685-90b4-7ae80a30ba54" (UID: "4a446110-2d66-4685-90b4-7ae80a30ba54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:51.046514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.046485    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-api-access-xh5zv" (OuterVolumeSpecName: "kube-api-access-xh5zv") pod "4a446110-2d66-4685-90b4-7ae80a30ba54" (UID: "4a446110-2d66-4685-90b4-7ae80a30ba54"). InnerVolumeSpecName "kube-api-access-xh5zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:51.049571 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.049545    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a446110-2d66-4685-90b4-7ae80a30ba54-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a446110-2d66-4685-90b4-7ae80a30ba54" (UID: "4a446110-2d66-4685-90b4-7ae80a30ba54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:51.145806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.145718    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a446110-2d66-4685-90b4-7ae80a30ba54-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:51.145806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.145750    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:51.145806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.145760    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a446110-2d66-4685-90b4-7ae80a30ba54-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:51.145806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.145769    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a446110-2d66-4685-90b4-7ae80a30ba54-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:51.145806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.145780    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh5zv\" (UniqueName: \"kubernetes.io/projected/4a446110-2d66-4685-90b4-7ae80a30ba54-kube-api-access-xh5zv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:23:51.833374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.833342    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"
Sep 17 19:23:51.853918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.853888    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"]
Sep 17 19:23:51.857297 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:51.857274    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-njj57"]
Sep 17 19:23:53.038194 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:53.038162    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" path="/var/lib/kubelet/pods/4a446110-2d66-4685-90b4-7ae80a30ba54/volumes"
Sep 17 19:23:54.607003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.606963    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"]
Sep 17 19:23:54.607410 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607391    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kserve-container"
Sep 17 19:23:54.607457 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607415    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kserve-container"
Sep 17 19:23:54.607457 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607430    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container"
Sep 17 19:23:54.607457 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607439    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container"
Sep 17 19:23:54.607568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607456    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kube-rbac-proxy"
Sep 17 19:23:54.607568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607467    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kube-rbac-proxy"
Sep 17 19:23:54.607568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607493    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="storage-initializer"
Sep 17 19:23:54.607568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607502    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="storage-initializer"
Sep 17 19:23:54.607568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607516    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="storage-initializer"
Sep 17 19:23:54.607568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607526    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="storage-initializer"
Sep 17 19:23:54.607568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607548    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy"
Sep 17 19:23:54.607568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607557    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy"
Sep 17 19:23:54.607990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607651    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kube-rbac-proxy"
Sep 17 19:23:54.607990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607665    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="45702a16-77b0-4fec-b3fa-ae67bae29592" containerName="kserve-container"
Sep 17 19:23:54.607990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607676    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kserve-container"
Sep 17 19:23:54.607990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.607687    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a446110-2d66-4685-90b4-7ae80a30ba54" containerName="kube-rbac-proxy"
Sep 17 19:23:54.635116 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.635083    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"]
Sep 17 19:23:54.635267 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.635210    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.637749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.637723    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 19:23:54.637880 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.637804    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 19:23:54.673089 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.673061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.673246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.673114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.673246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.673149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27e543e2-0760-4cad-a160-4eb93711a3ed-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.673246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.673202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e543e2-0760-4cad-a160-4eb93711a3ed-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.673246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.673218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8xn2\" (UniqueName: \"kubernetes.io/projected/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-api-access-f8xn2\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.774215 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.774171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.774381 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.774221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27e543e2-0760-4cad-a160-4eb93711a3ed-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.774381 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.774303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e543e2-0760-4cad-a160-4eb93711a3ed-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.774381 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.774327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8xn2\" (UniqueName: \"kubernetes.io/projected/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-api-access-f8xn2\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.774532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.774472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.774693 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.774676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27e543e2-0760-4cad-a160-4eb93711a3ed-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.774951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.774932    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.775039 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.775023    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.776911 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.776896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e543e2-0760-4cad-a160-4eb93711a3ed-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.783015 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.782991    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8xn2\" (UniqueName: \"kubernetes.io/projected/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-api-access-f8xn2\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-lp4fw\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:54.946005 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:54.945971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:23:55.236575 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:55.236543    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"]
Sep 17 19:23:55.239106 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:23:55.239074    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e543e2_0760_4cad_a160_4eb93711a3ed.slice/crio-1c21781cdf216ce46c4d5e5591d0ad24a4aa0af1929bff437123fdb53aa5c946 WatchSource:0}: Error finding container 1c21781cdf216ce46c4d5e5591d0ad24a4aa0af1929bff437123fdb53aa5c946: Status 404 returned error can't find the container with id 1c21781cdf216ce46c4d5e5591d0ad24a4aa0af1929bff437123fdb53aa5c946
Sep 17 19:23:55.855655 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:55.855604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" event={"ID":"27e543e2-0760-4cad-a160-4eb93711a3ed","Type":"ContainerStarted","Data":"62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129"}
Sep 17 19:23:55.856127 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:23:55.855684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" event={"ID":"27e543e2-0760-4cad-a160-4eb93711a3ed","Type":"ContainerStarted","Data":"1c21781cdf216ce46c4d5e5591d0ad24a4aa0af1929bff437123fdb53aa5c946"}
Sep 17 19:24:00.878400 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:00.878316    2583 generic.go:356] "Generic (PLEG): container finished" podID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerID="62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129" exitCode=0
Sep 17 19:24:00.878400 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:00.878367    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" event={"ID":"27e543e2-0760-4cad-a160-4eb93711a3ed","Type":"ContainerDied","Data":"62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129"}
Sep 17 19:24:01.885139 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:01.885105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" event={"ID":"27e543e2-0760-4cad-a160-4eb93711a3ed","Type":"ContainerStarted","Data":"2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58"}
Sep 17 19:24:01.885518 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:01.885168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" event={"ID":"27e543e2-0760-4cad-a160-4eb93711a3ed","Type":"ContainerStarted","Data":"c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc"}
Sep 17 19:24:01.885518 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:01.885422    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:24:01.907709 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:01.907658    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" podStartSLOduration=2.643925796 podStartE2EDuration="7.907641s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.263715204s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:24:01.905608123 +0000 UTC m=+1061.033349523" watchObservedRunningTime="2026-09-17 19:24:01.907641 +0000 UTC m=+1061.035382401"
Sep 17 19:24:02.891259 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:02.891223    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:24:07.751329 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:07.751282    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:24:08.903443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:08.903416    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:24:17.750875 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:17.750836    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:24:27.751751 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:27.751709    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:24:37.751123 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:37.751075    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 19:24:48.343253 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:48.343214    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:24:49.519440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:49.519400    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"]
Sep 17 19:24:49.519854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:49.519688    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kube-rbac-proxy" containerID="cri-o://3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5" gracePeriod=30
Sep 17 19:24:49.519854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:49.519674    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kserve-container" containerID="cri-o://439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478" gracePeriod=30
Sep 17 19:24:50.103582 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:50.103542    2583 generic.go:356] "Generic (PLEG): container finished" podID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerID="3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5" exitCode=2
Sep 17 19:24:50.103582 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:50.103584    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" event={"ID":"30590ad0-2511-4af8-bd1a-62a36d7221ee","Type":"ContainerDied","Data":"3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5"}
Sep 17 19:24:52.745588 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:52.745541    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 19:24:54.550487 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.550463    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:24:54.609550 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.609472    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30590ad0-2511-4af8-bd1a-62a36d7221ee-proxy-tls\") pod \"30590ad0-2511-4af8-bd1a-62a36d7221ee\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") "
Sep 17 19:24:54.609550 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.609501    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-rbac-proxy-sar-config\") pod \"30590ad0-2511-4af8-bd1a-62a36d7221ee\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") "
Sep 17 19:24:54.609550 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.609519    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-cabundle-cert\") pod \"30590ad0-2511-4af8-bd1a-62a36d7221ee\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") "
Sep 17 19:24:54.609550 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.609546    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30590ad0-2511-4af8-bd1a-62a36d7221ee-kserve-provision-location\") pod \"30590ad0-2511-4af8-bd1a-62a36d7221ee\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") "
Sep 17 19:24:54.609870 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.609608    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxxnj\" (UniqueName: \"kubernetes.io/projected/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-api-access-vxxnj\") pod \"30590ad0-2511-4af8-bd1a-62a36d7221ee\" (UID: \"30590ad0-2511-4af8-bd1a-62a36d7221ee\") "
Sep 17 19:24:54.609949 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.609927    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30590ad0-2511-4af8-bd1a-62a36d7221ee" (UID: "30590ad0-2511-4af8-bd1a-62a36d7221ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:24:54.609949 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.609934    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30590ad0-2511-4af8-bd1a-62a36d7221ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30590ad0-2511-4af8-bd1a-62a36d7221ee" (UID: "30590ad0-2511-4af8-bd1a-62a36d7221ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:54.610058 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.609924    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30590ad0-2511-4af8-bd1a-62a36d7221ee" (UID: "30590ad0-2511-4af8-bd1a-62a36d7221ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:24:54.611813 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.611782    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30590ad0-2511-4af8-bd1a-62a36d7221ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30590ad0-2511-4af8-bd1a-62a36d7221ee" (UID: "30590ad0-2511-4af8-bd1a-62a36d7221ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:24:54.611943 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.611871    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-api-access-vxxnj" (OuterVolumeSpecName: "kube-api-access-vxxnj") pod "30590ad0-2511-4af8-bd1a-62a36d7221ee" (UID: "30590ad0-2511-4af8-bd1a-62a36d7221ee"). InnerVolumeSpecName "kube-api-access-vxxnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:24:54.711345 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.711307    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30590ad0-2511-4af8-bd1a-62a36d7221ee-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:24:54.711345 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.711337    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:24:54.711345 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.711349    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30590ad0-2511-4af8-bd1a-62a36d7221ee-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:24:54.711577 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.711359    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30590ad0-2511-4af8-bd1a-62a36d7221ee-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:24:54.711577 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.711372    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxxnj\" (UniqueName: \"kubernetes.io/projected/30590ad0-2511-4af8-bd1a-62a36d7221ee-kube-api-access-vxxnj\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:24:54.725838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.725806    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"]
Sep 17 19:24:54.726161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.726149    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kube-rbac-proxy"
Sep 17 19:24:54.726211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.726163    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kube-rbac-proxy"
Sep 17 19:24:54.726211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.726186    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="storage-initializer"
Sep 17 19:24:54.726211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.726192    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="storage-initializer"
Sep 17 19:24:54.726211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.726200    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kserve-container"
Sep 17 19:24:54.726211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.726207    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kserve-container"
Sep 17 19:24:54.726388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.726264    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kube-rbac-proxy"
Sep 17 19:24:54.726388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.726275    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerName="kserve-container"
Sep 17 19:24:54.728517 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.728499    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.730780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.730757    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 19:24:54.730780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.730775    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 19:24:54.740909 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.740887    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"]
Sep 17 19:24:54.812493 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.812458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.812689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.812498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cfht\" (UniqueName: \"kubernetes.io/projected/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-api-access-9cfht\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.812689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.812569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26458828-3d42-4f9c-ad47-e0a88ca59fca-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.812689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.812607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26458828-3d42-4f9c-ad47-e0a88ca59fca-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.812689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.812660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.913665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.913551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.913665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.913597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cfht\" (UniqueName: \"kubernetes.io/projected/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-api-access-9cfht\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.913665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.913651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26458828-3d42-4f9c-ad47-e0a88ca59fca-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.913932 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.913685    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26458828-3d42-4f9c-ad47-e0a88ca59fca-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.913932 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.913707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.913932 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:24:54.913836    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 19:24:54.913932 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:24:54.913930    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26458828-3d42-4f9c-ad47-e0a88ca59fca-proxy-tls podName:26458828-3d42-4f9c-ad47-e0a88ca59fca nodeName:}" failed. No retries permitted until 2026-09-17 19:24:55.413904739 +0000 UTC m=+1114.541646130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/26458828-3d42-4f9c-ad47-e0a88ca59fca-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" (UID: "26458828-3d42-4f9c-ad47-e0a88ca59fca") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 19:24:54.914171 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.914147    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26458828-3d42-4f9c-ad47-e0a88ca59fca-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.914285 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.914267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.914324 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.914289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:54.922125 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:54.922094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cfht\" (UniqueName: \"kubernetes.io/projected/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-api-access-9cfht\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:55.126569 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.126537    2583 generic.go:356] "Generic (PLEG): container finished" podID="30590ad0-2511-4af8-bd1a-62a36d7221ee" containerID="439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478" exitCode=0
Sep 17 19:24:55.126744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.126602    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"
Sep 17 19:24:55.126744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.126640    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" event={"ID":"30590ad0-2511-4af8-bd1a-62a36d7221ee","Type":"ContainerDied","Data":"439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478"}
Sep 17 19:24:55.126744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.126716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh" event={"ID":"30590ad0-2511-4af8-bd1a-62a36d7221ee","Type":"ContainerDied","Data":"24029e083e1c078086c19a02b3511e29475fa20dfae6ada90ffe54cd36c34cc8"}
Sep 17 19:24:55.126848 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.126755    2583 scope.go:117] "RemoveContainer" containerID="3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5"
Sep 17 19:24:55.135139 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.135118    2583 scope.go:117] "RemoveContainer" containerID="439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478"
Sep 17 19:24:55.142598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.142580    2583 scope.go:117] "RemoveContainer" containerID="712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f"
Sep 17 19:24:55.143977 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.143956    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"]
Sep 17 19:24:55.147536 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.147510    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rmgdh"]
Sep 17 19:24:55.150537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.150521    2583 scope.go:117] "RemoveContainer" containerID="3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5"
Sep 17 19:24:55.150835 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:24:55.150816    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5\": container with ID starting with 3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5 not found: ID does not exist" containerID="3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5"
Sep 17 19:24:55.150888 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.150844    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5"} err="failed to get container status \"3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5\": rpc error: code = NotFound desc = could not find container \"3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5\": container with ID starting with 3985adbd4e27d7ee329a2e93cf17225c5caaf89575102437b04fbed43f9588f5 not found: ID does not exist"
Sep 17 19:24:55.150888 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.150862    2583 scope.go:117] "RemoveContainer" containerID="439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478"
Sep 17 19:24:55.151128 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:24:55.151113    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478\": container with ID starting with 439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478 not found: ID does not exist" containerID="439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478"
Sep 17 19:24:55.151179 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.151135    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478"} err="failed to get container status \"439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478\": rpc error: code = NotFound desc = could not find container \"439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478\": container with ID starting with 439d906f47adc834423b312fdefdb954689c1f0ca0827a0e16f6508a491d0478 not found: ID does not exist"
Sep 17 19:24:55.151179 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.151152    2583 scope.go:117] "RemoveContainer" containerID="712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f"
Sep 17 19:24:55.151374 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:24:55.151358    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f\": container with ID starting with 712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f not found: ID does not exist" containerID="712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f"
Sep 17 19:24:55.151417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.151378    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f"} err="failed to get container status \"712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f\": rpc error: code = NotFound desc = could not find container \"712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f\": container with ID starting with 712458136de6976f143b441426de56b7ae43e43039a755d35b982e7e4ab14d9f not found: ID does not exist"
Sep 17 19:24:55.417387 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.417358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26458828-3d42-4f9c-ad47-e0a88ca59fca-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:55.419923 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.419906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26458828-3d42-4f9c-ad47-e0a88ca59fca-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:55.637890 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.637853    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:24:55.911563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:55.911531    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"]
Sep 17 19:24:55.914228 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:24:55.914195    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26458828_3d42_4f9c_ad47_e0a88ca59fca.slice/crio-86e6e3d26ac661da6d4f0b714fd8a832c706aee65e7a8dbd22658daf576f4bb7 WatchSource:0}: Error finding container 86e6e3d26ac661da6d4f0b714fd8a832c706aee65e7a8dbd22658daf576f4bb7: Status 404 returned error can't find the container with id 86e6e3d26ac661da6d4f0b714fd8a832c706aee65e7a8dbd22658daf576f4bb7
Sep 17 19:24:56.133067 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:56.133027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" event={"ID":"26458828-3d42-4f9c-ad47-e0a88ca59fca","Type":"ContainerStarted","Data":"f992c5a4876b01d8038d5cd160c5e1da5fc0c01a12f10cfbd75444f1a8847456"}
Sep 17 19:24:56.133260 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:56.133101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" event={"ID":"26458828-3d42-4f9c-ad47-e0a88ca59fca","Type":"ContainerStarted","Data":"86e6e3d26ac661da6d4f0b714fd8a832c706aee65e7a8dbd22658daf576f4bb7"}
Sep 17 19:24:57.038191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:24:57.038162    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30590ad0-2511-4af8-bd1a-62a36d7221ee" path="/var/lib/kubelet/pods/30590ad0-2511-4af8-bd1a-62a36d7221ee/volumes"
Sep 17 19:25:00.153284 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:00.153251    2583 generic.go:356] "Generic (PLEG): container finished" podID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerID="f992c5a4876b01d8038d5cd160c5e1da5fc0c01a12f10cfbd75444f1a8847456" exitCode=0
Sep 17 19:25:00.153717 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:00.153324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" event={"ID":"26458828-3d42-4f9c-ad47-e0a88ca59fca","Type":"ContainerDied","Data":"f992c5a4876b01d8038d5cd160c5e1da5fc0c01a12f10cfbd75444f1a8847456"}
Sep 17 19:25:01.159810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:01.159773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" event={"ID":"26458828-3d42-4f9c-ad47-e0a88ca59fca","Type":"ContainerStarted","Data":"9f4064f25abc32da7d6fa6e1f1825c2afb0a5d2032f3fa606a32bee042022568"}
Sep 17 19:25:01.160216 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:01.159841    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" event={"ID":"26458828-3d42-4f9c-ad47-e0a88ca59fca","Type":"ContainerStarted","Data":"465b177c4bdc390045a4a0610899d837f49f21f36303c404cfcbe8609208fdf5"}
Sep 17 19:25:01.160216 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:01.160036    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:25:01.181725 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:01.181679    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" podStartSLOduration=3.178752882 podStartE2EDuration="7.181664167s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.002911285s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:25:01.179958062 +0000 UTC m=+1120.307699479" watchObservedRunningTime="2026-09-17 19:25:01.181664167 +0000 UTC m=+1120.309405568"
Sep 17 19:25:02.164398 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:02.164359    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:25:08.175685 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:08.175658    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:25:36.069214 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:36.069137    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:25:38.176947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:38.176905    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:25:47.075776 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:47.075734    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:25:48.177058 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:48.177021    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:25:51.145559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:51.145519    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"]
Sep 17 19:25:51.146009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:51.145853    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="kserve-container" containerID="cri-o://c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc" gracePeriod=30
Sep 17 19:25:51.146009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:51.145919    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="kube-rbac-proxy" containerID="cri-o://2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58" gracePeriod=30
Sep 17 19:25:51.379015 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:51.378979    2583 generic.go:356] "Generic (PLEG): container finished" podID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerID="2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58" exitCode=2
Sep 17 19:25:51.379289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:51.379048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" event={"ID":"27e543e2-0760-4cad-a160-4eb93711a3ed","Type":"ContainerDied","Data":"2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58"}
Sep 17 19:25:52.680013 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.679989    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:25:52.800925 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.800838    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-cabundle-cert\") pod \"27e543e2-0760-4cad-a160-4eb93711a3ed\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") "
Sep 17 19:25:52.800925 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.800893    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-rbac-proxy-sar-config\") pod \"27e543e2-0760-4cad-a160-4eb93711a3ed\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") "
Sep 17 19:25:52.801161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.800928    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8xn2\" (UniqueName: \"kubernetes.io/projected/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-api-access-f8xn2\") pod \"27e543e2-0760-4cad-a160-4eb93711a3ed\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") "
Sep 17 19:25:52.801161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.800973    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27e543e2-0760-4cad-a160-4eb93711a3ed-kserve-provision-location\") pod \"27e543e2-0760-4cad-a160-4eb93711a3ed\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") "
Sep 17 19:25:52.801161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.800997    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e543e2-0760-4cad-a160-4eb93711a3ed-proxy-tls\") pod \"27e543e2-0760-4cad-a160-4eb93711a3ed\" (UID: \"27e543e2-0760-4cad-a160-4eb93711a3ed\") "
Sep 17 19:25:52.801334 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.801247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27e543e2-0760-4cad-a160-4eb93711a3ed" (UID: "27e543e2-0760-4cad-a160-4eb93711a3ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:25:52.801388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.801336    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27e543e2-0760-4cad-a160-4eb93711a3ed" (UID: "27e543e2-0760-4cad-a160-4eb93711a3ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:25:52.803165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.803141    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-api-access-f8xn2" (OuterVolumeSpecName: "kube-api-access-f8xn2") pod "27e543e2-0760-4cad-a160-4eb93711a3ed" (UID: "27e543e2-0760-4cad-a160-4eb93711a3ed"). InnerVolumeSpecName "kube-api-access-f8xn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:25:52.803339 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.803320    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e543e2-0760-4cad-a160-4eb93711a3ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27e543e2-0760-4cad-a160-4eb93711a3ed" (UID: "27e543e2-0760-4cad-a160-4eb93711a3ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:25:52.805415 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.805391    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e543e2-0760-4cad-a160-4eb93711a3ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27e543e2-0760-4cad-a160-4eb93711a3ed" (UID: "27e543e2-0760-4cad-a160-4eb93711a3ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:25:52.901707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.901658    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8xn2\" (UniqueName: \"kubernetes.io/projected/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-api-access-f8xn2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:25:52.901707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.901700    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27e543e2-0760-4cad-a160-4eb93711a3ed-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:25:52.901707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.901711    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e543e2-0760-4cad-a160-4eb93711a3ed-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:25:52.901707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.901722    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:25:52.901707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:52.901729    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e543e2-0760-4cad-a160-4eb93711a3ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:25:53.390155 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.390121    2583 generic.go:356] "Generic (PLEG): container finished" podID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerID="c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc" exitCode=0
Sep 17 19:25:53.390340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.390173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" event={"ID":"27e543e2-0760-4cad-a160-4eb93711a3ed","Type":"ContainerDied","Data":"c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc"}
Sep 17 19:25:53.390340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.390191    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"
Sep 17 19:25:53.390340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.390237    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw" event={"ID":"27e543e2-0760-4cad-a160-4eb93711a3ed","Type":"ContainerDied","Data":"1c21781cdf216ce46c4d5e5591d0ad24a4aa0af1929bff437123fdb53aa5c946"}
Sep 17 19:25:53.390340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.390275    2583 scope.go:117] "RemoveContainer" containerID="2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58"
Sep 17 19:25:53.398665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.398645    2583 scope.go:117] "RemoveContainer" containerID="c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc"
Sep 17 19:25:53.406805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.406767    2583 scope.go:117] "RemoveContainer" containerID="62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129"
Sep 17 19:25:53.407889 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.407870    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"]
Sep 17 19:25:53.412573 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.412553    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-lp4fw"]
Sep 17 19:25:53.415229 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.415215    2583 scope.go:117] "RemoveContainer" containerID="2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58"
Sep 17 19:25:53.415513 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:25:53.415495    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58\": container with ID starting with 2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58 not found: ID does not exist" containerID="2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58"
Sep 17 19:25:53.415563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.415522    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58"} err="failed to get container status \"2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58\": rpc error: code = NotFound desc = could not find container \"2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58\": container with ID starting with 2082ea4984f5c694f239aac876eacfc0ae4a3e225146ab2b7a96a96953b16f58 not found: ID does not exist"
Sep 17 19:25:53.415563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.415544    2583 scope.go:117] "RemoveContainer" containerID="c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc"
Sep 17 19:25:53.415854 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:25:53.415837    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc\": container with ID starting with c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc not found: ID does not exist" containerID="c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc"
Sep 17 19:25:53.415896 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.415861    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc"} err="failed to get container status \"c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc\": rpc error: code = NotFound desc = could not find container \"c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc\": container with ID starting with c12cdc7fde0d8802a8398eeecc28069dc5573239f6aaf815fd378a9448398acc not found: ID does not exist"
Sep 17 19:25:53.415896 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.415876    2583 scope.go:117] "RemoveContainer" containerID="62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129"
Sep 17 19:25:53.416137 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:25:53.416120    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129\": container with ID starting with 62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129 not found: ID does not exist" containerID="62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129"
Sep 17 19:25:53.416178 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:53.416143    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129"} err="failed to get container status \"62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129\": rpc error: code = NotFound desc = could not find container \"62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129\": container with ID starting with 62899c3b7dd28996efb02c9ac83cf6cf6c123d6fd417f7946d76816f3194e129 not found: ID does not exist"
Sep 17 19:25:55.038903 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:55.038863    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" path="/var/lib/kubelet/pods/27e543e2-0760-4cad-a160-4eb93711a3ed/volumes"
Sep 17 19:25:56.351919 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.351883    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"]
Sep 17 19:25:56.352262 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.352234    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="kube-rbac-proxy"
Sep 17 19:25:56.352262 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.352249    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="kube-rbac-proxy"
Sep 17 19:25:56.352346 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.352267    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="kserve-container"
Sep 17 19:25:56.352346 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.352273    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="kserve-container"
Sep 17 19:25:56.352346 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.352281    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="storage-initializer"
Sep 17 19:25:56.352346 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.352287    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="storage-initializer"
Sep 17 19:25:56.352478 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.352358    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="kserve-container"
Sep 17 19:25:56.352478 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.352368    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e543e2-0760-4cad-a160-4eb93711a3ed" containerName="kube-rbac-proxy"
Sep 17 19:25:56.355717 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.355697    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.358043 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.358020    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 19:25:56.358165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.358026    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 19:25:56.365367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.365344    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"]
Sep 17 19:25:56.427882 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.427847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c8c24-838d-4219-ac20-4b038483ec5d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.427882 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.427882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.428099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.427960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.428099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.427982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5d24\" (UniqueName: \"kubernetes.io/projected/603c8c24-838d-4219-ac20-4b038483ec5d-kube-api-access-p5d24\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.428099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.428011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c8c24-838d-4219-ac20-4b038483ec5d-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.528876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.528838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c8c24-838d-4219-ac20-4b038483ec5d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.528876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.528874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.529053 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.528919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.529053 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.528935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5d24\" (UniqueName: \"kubernetes.io/projected/603c8c24-838d-4219-ac20-4b038483ec5d-kube-api-access-p5d24\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.529053 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.528950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c8c24-838d-4219-ac20-4b038483ec5d-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.529053 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:25:56.528980    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 19:25:56.529053 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:25:56.529053    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603c8c24-838d-4219-ac20-4b038483ec5d-proxy-tls podName:603c8c24-838d-4219-ac20-4b038483ec5d nodeName:}" failed. No retries permitted until 2026-09-17 19:25:57.029037008 +0000 UTC m=+1176.156778387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/603c8c24-838d-4219-ac20-4b038483ec5d-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" (UID: "603c8c24-838d-4219-ac20-4b038483ec5d") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 19:25:56.529354 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.529335    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c8c24-838d-4219-ac20-4b038483ec5d-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.529802 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.529783    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.529855 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.529784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:56.537512 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:56.537491    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5d24\" (UniqueName: \"kubernetes.io/projected/603c8c24-838d-4219-ac20-4b038483ec5d-kube-api-access-p5d24\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:57.032892 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:57.032852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c8c24-838d-4219-ac20-4b038483ec5d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:57.035580 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:57.035553    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c8c24-838d-4219-ac20-4b038483ec5d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:57.266797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:57.266754    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:25:57.540518 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:57.540492    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"]
Sep 17 19:25:57.543141 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:25:57.543110    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603c8c24_838d_4219_ac20_4b038483ec5d.slice/crio-cde85d28f4abde60d1cf3313f51f09b9ddc1daebad78e3ad0b7b7cd465e6977d WatchSource:0}: Error finding container cde85d28f4abde60d1cf3313f51f09b9ddc1daebad78e3ad0b7b7cd465e6977d: Status 404 returned error can't find the container with id cde85d28f4abde60d1cf3313f51f09b9ddc1daebad78e3ad0b7b7cd465e6977d
Sep 17 19:25:58.176821 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:58.176781    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:25:58.413912 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:58.413875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" event={"ID":"603c8c24-838d-4219-ac20-4b038483ec5d","Type":"ContainerStarted","Data":"fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b"}
Sep 17 19:25:58.414121 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:25:58.413946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" event={"ID":"603c8c24-838d-4219-ac20-4b038483ec5d","Type":"ContainerStarted","Data":"cde85d28f4abde60d1cf3313f51f09b9ddc1daebad78e3ad0b7b7cd465e6977d"}
Sep 17 19:26:08.176609 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:08.176565    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:26:15.034257 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:15.034211    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:26:22.996018 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:22.995989    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:26:22.997570 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:22.997547    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:26:25.548418 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:25.548379    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:26:27.543819 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:27.543788    2583 generic.go:356] "Generic (PLEG): container finished" podID="603c8c24-838d-4219-ac20-4b038483ec5d" containerID="fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b" exitCode=0
Sep 17 19:26:27.544294 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:27.543836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" event={"ID":"603c8c24-838d-4219-ac20-4b038483ec5d","Type":"ContainerDied","Data":"fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b"}
Sep 17 19:26:28.551313 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:28.551281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" event={"ID":"603c8c24-838d-4219-ac20-4b038483ec5d","Type":"ContainerStarted","Data":"d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0"}
Sep 17 19:26:28.551709 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:28.551371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" event={"ID":"603c8c24-838d-4219-ac20-4b038483ec5d","Type":"ContainerStarted","Data":"b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8"}
Sep 17 19:26:28.551709 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:28.551561    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:26:28.571957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:28.571912    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" podStartSLOduration=3.275349624 podStartE2EDuration="32.571897559s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.296547935s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:26:28.570527661 +0000 UTC m=+1207.698269063" watchObservedRunningTime="2026-09-17 19:26:28.571897559 +0000 UTC m=+1207.699638959"
Sep 17 19:26:29.555589 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:29.555555    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:26:35.567353 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:35.567325    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:26:35.569559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:35.567636    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"]
Sep 17 19:26:35.569559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:35.567864    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kserve-container" containerID="cri-o://465b177c4bdc390045a4a0610899d837f49f21f36303c404cfcbe8609208fdf5" gracePeriod=30
Sep 17 19:26:35.569559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:35.567897    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kube-rbac-proxy" containerID="cri-o://9f4064f25abc32da7d6fa6e1f1825c2afb0a5d2032f3fa606a32bee042022568" gracePeriod=30
Sep 17 19:26:36.592995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:36.592957    2583 generic.go:356] "Generic (PLEG): container finished" podID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerID="9f4064f25abc32da7d6fa6e1f1825c2afb0a5d2032f3fa606a32bee042022568" exitCode=2
Sep 17 19:26:36.593346 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:36.593025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" event={"ID":"26458828-3d42-4f9c-ad47-e0a88ca59fca","Type":"ContainerDied","Data":"9f4064f25abc32da7d6fa6e1f1825c2afb0a5d2032f3fa606a32bee042022568"}
Sep 17 19:26:38.170057 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:38.170018    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 19:26:41.618365 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.618278    2583 generic.go:356] "Generic (PLEG): container finished" podID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerID="465b177c4bdc390045a4a0610899d837f49f21f36303c404cfcbe8609208fdf5" exitCode=0
Sep 17 19:26:41.618365 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.618348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" event={"ID":"26458828-3d42-4f9c-ad47-e0a88ca59fca","Type":"ContainerDied","Data":"465b177c4bdc390045a4a0610899d837f49f21f36303c404cfcbe8609208fdf5"}
Sep 17 19:26:41.809398 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.809371    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:26:41.914290 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.914202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26458828-3d42-4f9c-ad47-e0a88ca59fca-proxy-tls\") pod \"26458828-3d42-4f9c-ad47-e0a88ca59fca\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") "
Sep 17 19:26:41.914290 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.914251    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26458828-3d42-4f9c-ad47-e0a88ca59fca-kserve-provision-location\") pod \"26458828-3d42-4f9c-ad47-e0a88ca59fca\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") "
Sep 17 19:26:41.914526 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.914294    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-rbac-proxy-sar-config\") pod \"26458828-3d42-4f9c-ad47-e0a88ca59fca\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") "
Sep 17 19:26:41.914526 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.914323    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cfht\" (UniqueName: \"kubernetes.io/projected/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-api-access-9cfht\") pod \"26458828-3d42-4f9c-ad47-e0a88ca59fca\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") "
Sep 17 19:26:41.914526 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.914345    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-cabundle-cert\") pod \"26458828-3d42-4f9c-ad47-e0a88ca59fca\" (UID: \"26458828-3d42-4f9c-ad47-e0a88ca59fca\") "
Sep 17 19:26:41.914738 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.914709    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26458828-3d42-4f9c-ad47-e0a88ca59fca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26458828-3d42-4f9c-ad47-e0a88ca59fca" (UID: "26458828-3d42-4f9c-ad47-e0a88ca59fca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:41.914792 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.914713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26458828-3d42-4f9c-ad47-e0a88ca59fca" (UID: "26458828-3d42-4f9c-ad47-e0a88ca59fca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:26:41.914832 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.914793    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26458828-3d42-4f9c-ad47-e0a88ca59fca" (UID: "26458828-3d42-4f9c-ad47-e0a88ca59fca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:26:41.916524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.916502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26458828-3d42-4f9c-ad47-e0a88ca59fca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26458828-3d42-4f9c-ad47-e0a88ca59fca" (UID: "26458828-3d42-4f9c-ad47-e0a88ca59fca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:26:41.916601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:41.916578    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-api-access-9cfht" (OuterVolumeSpecName: "kube-api-access-9cfht") pod "26458828-3d42-4f9c-ad47-e0a88ca59fca" (UID: "26458828-3d42-4f9c-ad47-e0a88ca59fca"). InnerVolumeSpecName "kube-api-access-9cfht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:26:42.015165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.015113    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cfht\" (UniqueName: \"kubernetes.io/projected/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-api-access-9cfht\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:26:42.015165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.015158    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:26:42.015165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.015169    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26458828-3d42-4f9c-ad47-e0a88ca59fca-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:26:42.015165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.015178    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26458828-3d42-4f9c-ad47-e0a88ca59fca-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:26:42.015165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.015189    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26458828-3d42-4f9c-ad47-e0a88ca59fca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:26:42.625342 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.625304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn" event={"ID":"26458828-3d42-4f9c-ad47-e0a88ca59fca","Type":"ContainerDied","Data":"86e6e3d26ac661da6d4f0b714fd8a832c706aee65e7a8dbd22658daf576f4bb7"}
Sep 17 19:26:42.625342 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.625335    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"
Sep 17 19:26:42.625860 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.625385    2583 scope.go:117] "RemoveContainer" containerID="9f4064f25abc32da7d6fa6e1f1825c2afb0a5d2032f3fa606a32bee042022568"
Sep 17 19:26:42.637679 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.637661    2583 scope.go:117] "RemoveContainer" containerID="465b177c4bdc390045a4a0610899d837f49f21f36303c404cfcbe8609208fdf5"
Sep 17 19:26:42.645230 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.645204    2583 scope.go:117] "RemoveContainer" containerID="f992c5a4876b01d8038d5cd160c5e1da5fc0c01a12f10cfbd75444f1a8847456"
Sep 17 19:26:42.647310 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.647289    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"]
Sep 17 19:26:42.651446 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:42.651421    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-zt8hn"]
Sep 17 19:26:43.039172 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:43.039142    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" path="/var/lib/kubelet/pods/26458828-3d42-4f9c-ad47-e0a88ca59fca/volumes"
Sep 17 19:26:58.301413 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.301377    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-406762d6"]
Sep 17 19:26:58.301853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.301714    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kube-rbac-proxy"
Sep 17 19:26:58.301853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.301725    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kube-rbac-proxy"
Sep 17 19:26:58.301853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.301733    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="storage-initializer"
Sep 17 19:26:58.301853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.301739    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="storage-initializer"
Sep 17 19:26:58.301853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.301747    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kserve-container"
Sep 17 19:26:58.301853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.301753    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kserve-container"
Sep 17 19:26:58.301853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.301810    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kube-rbac-proxy"
Sep 17 19:26:58.301853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.301818    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26458828-3d42-4f9c-ad47-e0a88ca59fca" containerName="kserve-container"
Sep 17 19:26:58.307233 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.307209    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-406762d6"
Sep 17 19:26:58.309175 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.309150    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-406762d6"]
Sep 17 19:26:58.456544 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.456510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwv2f\" (UniqueName: \"kubernetes.io/projected/7fd2cf80-ce7a-4824-ae46-b50737967f30-kube-api-access-wwv2f\") pod \"curl-metrics-406762d6\" (UID: \"7fd2cf80-ce7a-4824-ae46-b50737967f30\") " pod="kserve/curl-metrics-406762d6"
Sep 17 19:26:58.557949 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.557858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwv2f\" (UniqueName: \"kubernetes.io/projected/7fd2cf80-ce7a-4824-ae46-b50737967f30-kube-api-access-wwv2f\") pod \"curl-metrics-406762d6\" (UID: \"7fd2cf80-ce7a-4824-ae46-b50737967f30\") " pod="kserve/curl-metrics-406762d6"
Sep 17 19:26:58.566602 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.566579    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwv2f\" (UniqueName: \"kubernetes.io/projected/7fd2cf80-ce7a-4824-ae46-b50737967f30-kube-api-access-wwv2f\") pod \"curl-metrics-406762d6\" (UID: \"7fd2cf80-ce7a-4824-ae46-b50737967f30\") " pod="kserve/curl-metrics-406762d6"
Sep 17 19:26:58.628660 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.628632    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-406762d6"
Sep 17 19:26:58.898831 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.898805    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-406762d6"]
Sep 17 19:26:58.901298 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:26:58.901268    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fd2cf80_ce7a_4824_ae46_b50737967f30.slice/crio-bf37150270b5bb1367d5caea40eeec3842effe27780305b6944d1c9c7343f524 WatchSource:0}: Error finding container bf37150270b5bb1367d5caea40eeec3842effe27780305b6944d1c9c7343f524: Status 404 returned error can't find the container with id bf37150270b5bb1367d5caea40eeec3842effe27780305b6944d1c9c7343f524
Sep 17 19:26:58.903380 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:58.903359    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:26:59.704089 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:59.704064    2583 generic.go:356] "Generic (PLEG): container finished" podID="7fd2cf80-ce7a-4824-ae46-b50737967f30" containerID="77c40f50ef6b2c30085e16705f0f7907962828c373072ab16d7021b2b67986da" exitCode=0
Sep 17 19:26:59.704360 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:59.704143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-406762d6" event={"ID":"7fd2cf80-ce7a-4824-ae46-b50737967f30","Type":"ContainerDied","Data":"77c40f50ef6b2c30085e16705f0f7907962828c373072ab16d7021b2b67986da"}
Sep 17 19:26:59.704360 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:26:59.704185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-406762d6" event={"ID":"7fd2cf80-ce7a-4824-ae46-b50737967f30","Type":"ContainerStarted","Data":"bf37150270b5bb1367d5caea40eeec3842effe27780305b6944d1c9c7343f524"}
Sep 17 19:27:00.828441 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:00.828410    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-406762d6"
Sep 17 19:27:00.862083 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:00.862060    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-406762d6_7fd2cf80-ce7a-4824-ae46-b50737967f30/curl-metrics-406762d6/0.log"
Sep 17 19:27:00.885925 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:00.885901    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-406762d6"]
Sep 17 19:27:00.890096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:00.890078    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-406762d6"]
Sep 17 19:27:00.976800 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:00.976741    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwv2f\" (UniqueName: \"kubernetes.io/projected/7fd2cf80-ce7a-4824-ae46-b50737967f30-kube-api-access-wwv2f\") pod \"7fd2cf80-ce7a-4824-ae46-b50737967f30\" (UID: \"7fd2cf80-ce7a-4824-ae46-b50737967f30\") "
Sep 17 19:27:00.978867 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:00.978846    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd2cf80-ce7a-4824-ae46-b50737967f30-kube-api-access-wwv2f" (OuterVolumeSpecName: "kube-api-access-wwv2f") pod "7fd2cf80-ce7a-4824-ae46-b50737967f30" (UID: "7fd2cf80-ce7a-4824-ae46-b50737967f30"). InnerVolumeSpecName "kube-api-access-wwv2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:01.037955 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:01.037929    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd2cf80-ce7a-4824-ae46-b50737967f30" path="/var/lib/kubelet/pods/7fd2cf80-ce7a-4824-ae46-b50737967f30/volumes"
Sep 17 19:27:01.077629 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:01.077595    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwv2f\" (UniqueName: \"kubernetes.io/projected/7fd2cf80-ce7a-4824-ae46-b50737967f30-kube-api-access-wwv2f\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:27:01.713536 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:01.713509    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-406762d6"
Sep 17 19:27:01.713745 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:01.713553    2583 scope.go:117] "RemoveContainer" containerID="77c40f50ef6b2c30085e16705f0f7907962828c373072ab16d7021b2b67986da"
Sep 17 19:27:06.274355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.274323    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-9d2b3bfe"]
Sep 17 19:27:06.274732 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.274705    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd2cf80-ce7a-4824-ae46-b50737967f30" containerName="curl-metrics-406762d6"
Sep 17 19:27:06.274732 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.274718    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd2cf80-ce7a-4824-ae46-b50737967f30" containerName="curl-metrics-406762d6"
Sep 17 19:27:06.274822 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.274793    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd2cf80-ce7a-4824-ae46-b50737967f30" containerName="curl-metrics-406762d6"
Sep 17 19:27:06.279006 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.278989    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9d2b3bfe"
Sep 17 19:27:06.283196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.283170    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-9d2b3bfe"]
Sep 17 19:27:06.421935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.421902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4c2p\" (UniqueName: \"kubernetes.io/projected/d4b32a46-54a1-499a-a4a3-13cbac68e6c4-kube-api-access-f4c2p\") pod \"curl-metrics-9d2b3bfe\" (UID: \"d4b32a46-54a1-499a-a4a3-13cbac68e6c4\") " pod="kserve/curl-metrics-9d2b3bfe"
Sep 17 19:27:06.522632 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.522590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4c2p\" (UniqueName: \"kubernetes.io/projected/d4b32a46-54a1-499a-a4a3-13cbac68e6c4-kube-api-access-f4c2p\") pod \"curl-metrics-9d2b3bfe\" (UID: \"d4b32a46-54a1-499a-a4a3-13cbac68e6c4\") " pod="kserve/curl-metrics-9d2b3bfe"
Sep 17 19:27:06.530888 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.530818    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4c2p\" (UniqueName: \"kubernetes.io/projected/d4b32a46-54a1-499a-a4a3-13cbac68e6c4-kube-api-access-f4c2p\") pod \"curl-metrics-9d2b3bfe\" (UID: \"d4b32a46-54a1-499a-a4a3-13cbac68e6c4\") " pod="kserve/curl-metrics-9d2b3bfe"
Sep 17 19:27:06.604628 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.604585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9d2b3bfe"
Sep 17 19:27:06.873401 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:06.873377    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-9d2b3bfe"]
Sep 17 19:27:06.876047 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:27:06.876017    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b32a46_54a1_499a_a4a3_13cbac68e6c4.slice/crio-b271554852c65298b6bc7bccbff385f818b3d5071f607c94e10a520106dd22cf WatchSource:0}: Error finding container b271554852c65298b6bc7bccbff385f818b3d5071f607c94e10a520106dd22cf: Status 404 returned error can't find the container with id b271554852c65298b6bc7bccbff385f818b3d5071f607c94e10a520106dd22cf
Sep 17 19:27:07.741328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:07.741295    2583 generic.go:356] "Generic (PLEG): container finished" podID="d4b32a46-54a1-499a-a4a3-13cbac68e6c4" containerID="eb8ebad4d602a8df8daa2c0215f14a4aabf0f8d55fa6aee662ef95339ad7f050" exitCode=0
Sep 17 19:27:07.741741 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:07.741393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-9d2b3bfe" event={"ID":"d4b32a46-54a1-499a-a4a3-13cbac68e6c4","Type":"ContainerDied","Data":"eb8ebad4d602a8df8daa2c0215f14a4aabf0f8d55fa6aee662ef95339ad7f050"}
Sep 17 19:27:07.741741 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:07.741445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-9d2b3bfe" event={"ID":"d4b32a46-54a1-499a-a4a3-13cbac68e6c4","Type":"ContainerStarted","Data":"b271554852c65298b6bc7bccbff385f818b3d5071f607c94e10a520106dd22cf"}
Sep 17 19:27:08.873460 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:08.873436    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9d2b3bfe"
Sep 17 19:27:08.892512 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:08.892492    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-9d2b3bfe_d4b32a46-54a1-499a-a4a3-13cbac68e6c4/curl-metrics-9d2b3bfe/0.log"
Sep 17 19:27:08.914666 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:08.914632    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-9d2b3bfe"]
Sep 17 19:27:08.916724 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:08.916701    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-9d2b3bfe"]
Sep 17 19:27:09.044283 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:09.044211    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4c2p\" (UniqueName: \"kubernetes.io/projected/d4b32a46-54a1-499a-a4a3-13cbac68e6c4-kube-api-access-f4c2p\") pod \"d4b32a46-54a1-499a-a4a3-13cbac68e6c4\" (UID: \"d4b32a46-54a1-499a-a4a3-13cbac68e6c4\") "
Sep 17 19:27:09.046333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:09.046313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b32a46-54a1-499a-a4a3-13cbac68e6c4-kube-api-access-f4c2p" (OuterVolumeSpecName: "kube-api-access-f4c2p") pod "d4b32a46-54a1-499a-a4a3-13cbac68e6c4" (UID: "d4b32a46-54a1-499a-a4a3-13cbac68e6c4"). InnerVolumeSpecName "kube-api-access-f4c2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:09.145130 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:09.145094    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4c2p\" (UniqueName: \"kubernetes.io/projected/d4b32a46-54a1-499a-a4a3-13cbac68e6c4-kube-api-access-f4c2p\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.751436 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:09.751406    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-9d2b3bfe"
Sep 17 19:27:09.751644 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:09.751406    2583 scope.go:117] "RemoveContainer" containerID="eb8ebad4d602a8df8daa2c0215f14a4aabf0f8d55fa6aee662ef95339ad7f050"
Sep 17 19:27:11.042710 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:11.042675    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b32a46-54a1-499a-a4a3-13cbac68e6c4" path="/var/lib/kubelet/pods/d4b32a46-54a1-499a-a4a3-13cbac68e6c4/volumes"
Sep 17 19:27:14.300460 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.300419    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"]
Sep 17 19:27:14.300874 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.300793    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b32a46-54a1-499a-a4a3-13cbac68e6c4" containerName="curl-metrics-9d2b3bfe"
Sep 17 19:27:14.300874 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.300806    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b32a46-54a1-499a-a4a3-13cbac68e6c4" containerName="curl-metrics-9d2b3bfe"
Sep 17 19:27:14.300874 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.300856    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b32a46-54a1-499a-a4a3-13cbac68e6c4" containerName="curl-metrics-9d2b3bfe"
Sep 17 19:27:14.304967 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.304948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.307870 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.307848    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 19:27:14.308409 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.308395    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 19:27:14.327526 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.327499    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"]
Sep 17 19:27:14.486814 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.486790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-cabundle-cert\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.487000 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.486840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9774bac4-dc6d-498f-8a76-edcada37d194-kserve-provision-location\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.487000 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.486903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qht9v\" (UniqueName: \"kubernetes.io/projected/9774bac4-dc6d-498f-8a76-edcada37d194-kube-api-access-qht9v\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.487000 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.486942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9774bac4-dc6d-498f-8a76-edcada37d194-proxy-tls\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.487130 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.486998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.588197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.588162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9774bac4-dc6d-498f-8a76-edcada37d194-kserve-provision-location\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.588388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.588215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qht9v\" (UniqueName: \"kubernetes.io/projected/9774bac4-dc6d-498f-8a76-edcada37d194-kube-api-access-qht9v\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.588388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.588256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9774bac4-dc6d-498f-8a76-edcada37d194-proxy-tls\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.588388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.588299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.588388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.588331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-cabundle-cert\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.588695 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.588667    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9774bac4-dc6d-498f-8a76-edcada37d194-kserve-provision-location\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.588924 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.588905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.589023 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.589000    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-cabundle-cert\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.590969 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.590947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9774bac4-dc6d-498f-8a76-edcada37d194-proxy-tls\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.597074 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.597054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qht9v\" (UniqueName: \"kubernetes.io/projected/9774bac4-dc6d-498f-8a76-edcada37d194-kube-api-access-qht9v\") pod \"isvc-sklearn-predictor-bb6f97488-qtq44\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.614855 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.614835    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:14.887762 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:14.887733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"]
Sep 17 19:27:14.890183 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:27:14.890159    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9774bac4_dc6d_498f_8a76_edcada37d194.slice/crio-eee1e0df21375320d91f4a26ec3aa88b2a37c4639462fd026563507bf1a6ae18 WatchSource:0}: Error finding container eee1e0df21375320d91f4a26ec3aa88b2a37c4639462fd026563507bf1a6ae18: Status 404 returned error can't find the container with id eee1e0df21375320d91f4a26ec3aa88b2a37c4639462fd026563507bf1a6ae18
Sep 17 19:27:15.781185 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:15.781145    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" event={"ID":"9774bac4-dc6d-498f-8a76-edcada37d194","Type":"ContainerStarted","Data":"e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91"}
Sep 17 19:27:15.781569 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:15.781204    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" event={"ID":"9774bac4-dc6d-498f-8a76-edcada37d194","Type":"ContainerStarted","Data":"eee1e0df21375320d91f4a26ec3aa88b2a37c4639462fd026563507bf1a6ae18"}
Sep 17 19:27:19.800712 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:19.800672    2583 generic.go:356] "Generic (PLEG): container finished" podID="9774bac4-dc6d-498f-8a76-edcada37d194" containerID="e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91" exitCode=0
Sep 17 19:27:19.801189 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:19.800729    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" event={"ID":"9774bac4-dc6d-498f-8a76-edcada37d194","Type":"ContainerDied","Data":"e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91"}
Sep 17 19:27:20.808035 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:20.808001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" event={"ID":"9774bac4-dc6d-498f-8a76-edcada37d194","Type":"ContainerStarted","Data":"243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5"}
Sep 17 19:27:20.808390 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:20.808067    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" event={"ID":"9774bac4-dc6d-498f-8a76-edcada37d194","Type":"ContainerStarted","Data":"c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd"}
Sep 17 19:27:20.808390 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:20.808329    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:20.808390 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:20.808377    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:20.809812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:20.809786    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:20.830395 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:20.830334    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podStartSLOduration=2.83897653 podStartE2EDuration="6.830315137s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.991338607s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:20.829864072 +0000 UTC m=+1259.957605474" watchObservedRunningTime="2026-09-17 19:27:20.830315137 +0000 UTC m=+1259.958056542"
Sep 17 19:27:21.813028 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:21.812986    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:23.053837 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:23.053805    2583 scope.go:117] "RemoveContainer" containerID="cef7a564f529e29d50dbb1b64180030518b390222424a6b285bc1ba0f776ad41"
Sep 17 19:27:26.818844 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:26.818811    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:27:26.819324 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:26.819300    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:36.820284 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:36.820245    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:46.819287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:46.819249    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:56.819420 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:27:56.819382    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:28:00.068221 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:00.068191    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:28:06.819245 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:06.819201    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:28:11.075842 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:11.075796    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:28:13.321987 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:13.321954    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"]
Sep 17 19:28:13.322409 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:13.322231    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="kserve-container" containerID="cri-o://b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8" gracePeriod=30
Sep 17 19:28:13.322409 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:13.322286    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="kube-rbac-proxy" containerID="cri-o://d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0" gracePeriod=30
Sep 17 19:28:14.044171 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.044135    2583 generic.go:356] "Generic (PLEG): container finished" podID="603c8c24-838d-4219-ac20-4b038483ec5d" containerID="d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0" exitCode=2
Sep 17 19:28:14.044350 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.044191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" event={"ID":"603c8c24-838d-4219-ac20-4b038483ec5d","Type":"ContainerDied","Data":"d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0"}
Sep 17 19:28:14.858077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.858056    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:28:14.911862 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.911831    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c8c24-838d-4219-ac20-4b038483ec5d-kserve-provision-location\") pod \"603c8c24-838d-4219-ac20-4b038483ec5d\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") "
Sep 17 19:28:14.911862 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.911865    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-cabundle-cert\") pod \"603c8c24-838d-4219-ac20-4b038483ec5d\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") "
Sep 17 19:28:14.912096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.911908    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c8c24-838d-4219-ac20-4b038483ec5d-proxy-tls\") pod \"603c8c24-838d-4219-ac20-4b038483ec5d\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") "
Sep 17 19:28:14.912096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.911924    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5d24\" (UniqueName: \"kubernetes.io/projected/603c8c24-838d-4219-ac20-4b038483ec5d-kube-api-access-p5d24\") pod \"603c8c24-838d-4219-ac20-4b038483ec5d\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") "
Sep 17 19:28:14.912096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.912058    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-kube-rbac-proxy-sar-config\") pod \"603c8c24-838d-4219-ac20-4b038483ec5d\" (UID: \"603c8c24-838d-4219-ac20-4b038483ec5d\") "
Sep 17 19:28:14.912283 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.912254    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "603c8c24-838d-4219-ac20-4b038483ec5d" (UID: "603c8c24-838d-4219-ac20-4b038483ec5d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:28:14.912422 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.912362    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:28:14.912422 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.912389    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "603c8c24-838d-4219-ac20-4b038483ec5d" (UID: "603c8c24-838d-4219-ac20-4b038483ec5d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:28:14.914248 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.914196    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603c8c24-838d-4219-ac20-4b038483ec5d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "603c8c24-838d-4219-ac20-4b038483ec5d" (UID: "603c8c24-838d-4219-ac20-4b038483ec5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:28:14.914248 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.914199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603c8c24-838d-4219-ac20-4b038483ec5d-kube-api-access-p5d24" (OuterVolumeSpecName: "kube-api-access-p5d24") pod "603c8c24-838d-4219-ac20-4b038483ec5d" (UID: "603c8c24-838d-4219-ac20-4b038483ec5d"). InnerVolumeSpecName "kube-api-access-p5d24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:28:14.916474 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:14.916439    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603c8c24-838d-4219-ac20-4b038483ec5d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "603c8c24-838d-4219-ac20-4b038483ec5d" (UID: "603c8c24-838d-4219-ac20-4b038483ec5d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:28:15.012878 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.012841    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c8c24-838d-4219-ac20-4b038483ec5d-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.012878 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.012875    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5d24\" (UniqueName: \"kubernetes.io/projected/603c8c24-838d-4219-ac20-4b038483ec5d-kube-api-access-p5d24\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.012878 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.012887    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c8c24-838d-4219-ac20-4b038483ec5d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.013118 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.012896    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c8c24-838d-4219-ac20-4b038483ec5d-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.050832 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.050796    2583 generic.go:356] "Generic (PLEG): container finished" podID="603c8c24-838d-4219-ac20-4b038483ec5d" containerID="b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8" exitCode=0
Sep 17 19:28:15.050958 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.050847    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" event={"ID":"603c8c24-838d-4219-ac20-4b038483ec5d","Type":"ContainerDied","Data":"b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8"}
Sep 17 19:28:15.050958 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.050904    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc" event={"ID":"603c8c24-838d-4219-ac20-4b038483ec5d","Type":"ContainerDied","Data":"cde85d28f4abde60d1cf3313f51f09b9ddc1daebad78e3ad0b7b7cd465e6977d"}
Sep 17 19:28:15.050958 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.050911    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"
Sep 17 19:28:15.050958 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.050931    2583 scope.go:117] "RemoveContainer" containerID="d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0"
Sep 17 19:28:15.059509 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.059486    2583 scope.go:117] "RemoveContainer" containerID="b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8"
Sep 17 19:28:15.067821 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.067800    2583 scope.go:117] "RemoveContainer" containerID="fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b"
Sep 17 19:28:15.068811 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.068700    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"]
Sep 17 19:28:15.072980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.072957    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-pbtjc"]
Sep 17 19:28:15.076253 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.076237    2583 scope.go:117] "RemoveContainer" containerID="d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0"
Sep 17 19:28:15.076548 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:28:15.076530    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0\": container with ID starting with d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0 not found: ID does not exist" containerID="d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0"
Sep 17 19:28:15.076605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.076556    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0"} err="failed to get container status \"d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0\": rpc error: code = NotFound desc = could not find container \"d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0\": container with ID starting with d6c35efedf7d82c306d6bc8676427712b2ed65a61e972c3b23e5af539821ecd0 not found: ID does not exist"
Sep 17 19:28:15.076605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.076573    2583 scope.go:117] "RemoveContainer" containerID="b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8"
Sep 17 19:28:15.076849 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:28:15.076833    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8\": container with ID starting with b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8 not found: ID does not exist" containerID="b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8"
Sep 17 19:28:15.076891 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.076855    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8"} err="failed to get container status \"b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8\": rpc error: code = NotFound desc = could not find container \"b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8\": container with ID starting with b49c8a1c25d624cecf214e4a57a17336bf462051e9271452458dfd4173e6f6a8 not found: ID does not exist"
Sep 17 19:28:15.076891 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.076869    2583 scope.go:117] "RemoveContainer" containerID="fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b"
Sep 17 19:28:15.077101 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:28:15.077084    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b\": container with ID starting with fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b not found: ID does not exist" containerID="fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b"
Sep 17 19:28:15.077142 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:15.077104    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b"} err="failed to get container status \"fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b\": rpc error: code = NotFound desc = could not find container \"fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b\": container with ID starting with fd72001f7f5aea23eaaafefc81b128e172669b49feb79c5ff0761c83c8a1165b not found: ID does not exist"
Sep 17 19:28:16.819378 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:16.819342    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:28:17.038548 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:17.038500    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" path="/var/lib/kubelet/pods/603c8c24-838d-4219-ac20-4b038483ec5d/volumes"
Sep 17 19:28:18.527064 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.527031    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"]
Sep 17 19:28:18.527551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.527355    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="kube-rbac-proxy"
Sep 17 19:28:18.527551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.527367    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="kube-rbac-proxy"
Sep 17 19:28:18.527551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.527378    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="storage-initializer"
Sep 17 19:28:18.527551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.527384    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="storage-initializer"
Sep 17 19:28:18.527551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.527393    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="kserve-container"
Sep 17 19:28:18.527551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.527398    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="kserve-container"
Sep 17 19:28:18.527551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.527443    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="kserve-container"
Sep 17 19:28:18.527551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.527452    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="603c8c24-838d-4219-ac20-4b038483ec5d" containerName="kube-rbac-proxy"
Sep 17 19:28:18.532367 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.532351    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.534717 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.534689    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 19:28:18.534827 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.534739    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 19:28:18.541475 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.541451    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"]
Sep 17 19:28:18.640641 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.640575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.640830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.640688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97808097-7481-479a-9eb4-84c4154ef102-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.640830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.640712    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d59hq\" (UniqueName: \"kubernetes.io/projected/97808097-7481-479a-9eb4-84c4154ef102-kube-api-access-d59hq\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.640830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.640737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97808097-7481-479a-9eb4-84c4154ef102-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.640830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.640803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.741832 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.741801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.742022 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.741856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97808097-7481-479a-9eb4-84c4154ef102-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.742022 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.741876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59hq\" (UniqueName: \"kubernetes.io/projected/97808097-7481-479a-9eb4-84c4154ef102-kube-api-access-d59hq\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.742022 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.741895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97808097-7481-479a-9eb4-84c4154ef102-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.742022 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.741936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.742022 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:28:18.741983    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 19:28:18.742312 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:28:18.742118    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97808097-7481-479a-9eb4-84c4154ef102-proxy-tls podName:97808097-7481-479a-9eb4-84c4154ef102 nodeName:}" failed. No retries permitted until 2026-09-17 19:28:19.242097449 +0000 UTC m=+1318.369838829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/97808097-7481-479a-9eb4-84c4154ef102-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" (UID: "97808097-7481-479a-9eb4-84c4154ef102") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 19:28:18.742379 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.742351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97808097-7481-479a-9eb4-84c4154ef102-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.742522 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.742505    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.742601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.742585    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:18.753959 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:18.753937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59hq\" (UniqueName: \"kubernetes.io/projected/97808097-7481-479a-9eb4-84c4154ef102-kube-api-access-d59hq\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:19.246556 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:19.246520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97808097-7481-479a-9eb4-84c4154ef102-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:19.249138 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:19.249106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97808097-7481-479a-9eb4-84c4154ef102-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:19.443168 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:19.443128    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:19.722155 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:19.722131    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"]
Sep 17 19:28:19.723846 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:28:19.723820    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97808097_7481_479a_9eb4_84c4154ef102.slice/crio-0b88ebf04799b6c26b9920b41f561fe13071702b0e12445e68a444d4126dbd19 WatchSource:0}: Error finding container 0b88ebf04799b6c26b9920b41f561fe13071702b0e12445e68a444d4126dbd19: Status 404 returned error can't find the container with id 0b88ebf04799b6c26b9920b41f561fe13071702b0e12445e68a444d4126dbd19
Sep 17 19:28:20.073894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:20.073802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" event={"ID":"97808097-7481-479a-9eb4-84c4154ef102","Type":"ContainerStarted","Data":"7ed59716bc8c5fc946bf8853c0d11fd36301bae5c6da3db147eab5991c113dd9"}
Sep 17 19:28:20.073894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:20.073860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" event={"ID":"97808097-7481-479a-9eb4-84c4154ef102","Type":"ContainerStarted","Data":"0b88ebf04799b6c26b9920b41f561fe13071702b0e12445e68a444d4126dbd19"}
Sep 17 19:28:25.096304 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:25.096273    2583 generic.go:356] "Generic (PLEG): container finished" podID="97808097-7481-479a-9eb4-84c4154ef102" containerID="7ed59716bc8c5fc946bf8853c0d11fd36301bae5c6da3db147eab5991c113dd9" exitCode=0
Sep 17 19:28:25.096709 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:25.096346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" event={"ID":"97808097-7481-479a-9eb4-84c4154ef102","Type":"ContainerDied","Data":"7ed59716bc8c5fc946bf8853c0d11fd36301bae5c6da3db147eab5991c113dd9"}
Sep 17 19:28:26.103768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:26.103732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" event={"ID":"97808097-7481-479a-9eb4-84c4154ef102","Type":"ContainerStarted","Data":"3d967a7517441e4bd4f3654ad644311e19826b6636f4a50a50a1075a1fea358d"}
Sep 17 19:28:26.104234 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:26.103792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" event={"ID":"97808097-7481-479a-9eb4-84c4154ef102","Type":"ContainerStarted","Data":"a81af3335b994d101d734a9275ac8e9ea744efc8ffb7973ba6b519a8999ffd28"}
Sep 17 19:28:26.104234 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:26.103978    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:26.124818 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:26.124754    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" podStartSLOduration=2.995948653 podStartE2EDuration="8.124734309s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.128785656s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:26.122798594 +0000 UTC m=+1325.250539996" watchObservedRunningTime="2026-09-17 19:28:26.124734309 +0000 UTC m=+1325.252475711"
Sep 17 19:28:26.819421 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:26.819381    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:28:27.107952 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:27.107877    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:33.120271 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:33.120189    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:28:36.819765 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:36.819736    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:28:44.707889 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:44.707844    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:28:46.717112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:46.717080    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:28:48.726932 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:48.726899    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:28:50.737910 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:50.737880    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:28:52.748314 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:52.748267    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:28:54.757610 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:54.757565    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:28:56.767656 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:56.767591    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:28:58.777497 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:28:58.777466    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:29:00.790373 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:00.790345    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:29:02.802487 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:02.802452    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:29:04.812263 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:04.812233    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:29:06.822170 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:06.822134    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:29:08.832030 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:08.831991    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:29:10.844005 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:10.843973    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:29:12.853958 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:12.853934    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:29:14.864302 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:14.864274    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-qtq44_9774bac4-dc6d-498f-8a76-edcada37d194/kserve-container/0.log"
Sep 17 19:29:14.991015 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:14.990975    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"]
Sep 17 19:29:14.991258 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:14.991236    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" containerID="cri-o://c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd" gracePeriod=30
Sep 17 19:29:14.991343 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:14.991281    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kube-rbac-proxy" containerID="cri-o://243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5" gracePeriod=30
Sep 17 19:29:15.316281 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:15.316247    2583 generic.go:356] "Generic (PLEG): container finished" podID="9774bac4-dc6d-498f-8a76-edcada37d194" containerID="243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5" exitCode=2
Sep 17 19:29:15.316468 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:15.316319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" event={"ID":"9774bac4-dc6d-498f-8a76-edcada37d194","Type":"ContainerDied","Data":"243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5"}
Sep 17 19:29:16.814036 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:16.813996    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 19:29:16.820090 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:16.820063    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:29:19.934747 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:19.934720    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:29:20.071983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.071891    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9774bac4-dc6d-498f-8a76-edcada37d194-kserve-provision-location\") pod \"9774bac4-dc6d-498f-8a76-edcada37d194\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") "
Sep 17 19:29:20.071983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.071940    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-kube-rbac-proxy-sar-config\") pod \"9774bac4-dc6d-498f-8a76-edcada37d194\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") "
Sep 17 19:29:20.071983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.071979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-cabundle-cert\") pod \"9774bac4-dc6d-498f-8a76-edcada37d194\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") "
Sep 17 19:29:20.072279 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.072040    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qht9v\" (UniqueName: \"kubernetes.io/projected/9774bac4-dc6d-498f-8a76-edcada37d194-kube-api-access-qht9v\") pod \"9774bac4-dc6d-498f-8a76-edcada37d194\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") "
Sep 17 19:29:20.072279 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.072064    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9774bac4-dc6d-498f-8a76-edcada37d194-proxy-tls\") pod \"9774bac4-dc6d-498f-8a76-edcada37d194\" (UID: \"9774bac4-dc6d-498f-8a76-edcada37d194\") "
Sep 17 19:29:20.072279 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.072236    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774bac4-dc6d-498f-8a76-edcada37d194-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9774bac4-dc6d-498f-8a76-edcada37d194" (UID: "9774bac4-dc6d-498f-8a76-edcada37d194"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:20.072434 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.072360    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9774bac4-dc6d-498f-8a76-edcada37d194" (UID: "9774bac4-dc6d-498f-8a76-edcada37d194"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:29:20.072434 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.072380    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9774bac4-dc6d-498f-8a76-edcada37d194" (UID: "9774bac4-dc6d-498f-8a76-edcada37d194"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:29:20.074339 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.074314    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9774bac4-dc6d-498f-8a76-edcada37d194-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9774bac4-dc6d-498f-8a76-edcada37d194" (UID: "9774bac4-dc6d-498f-8a76-edcada37d194"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:29:20.074434 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.074419    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9774bac4-dc6d-498f-8a76-edcada37d194-kube-api-access-qht9v" (OuterVolumeSpecName: "kube-api-access-qht9v") pod "9774bac4-dc6d-498f-8a76-edcada37d194" (UID: "9774bac4-dc6d-498f-8a76-edcada37d194"). InnerVolumeSpecName "kube-api-access-qht9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:29:20.173269 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.173236    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qht9v\" (UniqueName: \"kubernetes.io/projected/9774bac4-dc6d-498f-8a76-edcada37d194-kube-api-access-qht9v\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:29:20.173269 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.173261    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9774bac4-dc6d-498f-8a76-edcada37d194-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:29:20.173269 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.173270    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9774bac4-dc6d-498f-8a76-edcada37d194-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:29:20.173269 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.173279    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:29:20.173579 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.173292    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9774bac4-dc6d-498f-8a76-edcada37d194-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:29:20.182386 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.182358    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"]
Sep 17 19:29:20.182720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.182706    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kube-rbac-proxy"
Sep 17 19:29:20.182770 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.182722    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kube-rbac-proxy"
Sep 17 19:29:20.182770 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.182731    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="storage-initializer"
Sep 17 19:29:20.182770 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.182736    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="storage-initializer"
Sep 17 19:29:20.182770 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.182754    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container"
Sep 17 19:29:20.182770 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.182759    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container"
Sep 17 19:29:20.182947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.182802    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kube-rbac-proxy"
Sep 17 19:29:20.182947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.182811    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" containerName="kserve-container"
Sep 17 19:29:20.184886 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.184872    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.186986 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.186965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 19:29:20.187086 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.187017    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 19:29:20.196165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.196139    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"]
Sep 17 19:29:20.273837 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.273802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eabc9730-35e2-4947-b915-c7da39f13f5f-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.274002 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.273856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.274002 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.273878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eabc9730-35e2-4947-b915-c7da39f13f5f-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.274002 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.273923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5c87\" (UniqueName: \"kubernetes.io/projected/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-api-access-w5c87\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.274002 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.273940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.343300 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.343209    2583 generic.go:356] "Generic (PLEG): container finished" podID="9774bac4-dc6d-498f-8a76-edcada37d194" containerID="c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd" exitCode=0
Sep 17 19:29:20.343300 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.343292    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"
Sep 17 19:29:20.343516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.343295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" event={"ID":"9774bac4-dc6d-498f-8a76-edcada37d194","Type":"ContainerDied","Data":"c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd"}
Sep 17 19:29:20.343516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.343386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44" event={"ID":"9774bac4-dc6d-498f-8a76-edcada37d194","Type":"ContainerDied","Data":"eee1e0df21375320d91f4a26ec3aa88b2a37c4639462fd026563507bf1a6ae18"}
Sep 17 19:29:20.343516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.343420    2583 scope.go:117] "RemoveContainer" containerID="243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5"
Sep 17 19:29:20.352864 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.352848    2583 scope.go:117] "RemoveContainer" containerID="c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd"
Sep 17 19:29:20.360326 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.360304    2583 scope.go:117] "RemoveContainer" containerID="e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91"
Sep 17 19:29:20.366009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.365987    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"]
Sep 17 19:29:20.368853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.368836    2583 scope.go:117] "RemoveContainer" containerID="243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5"
Sep 17 19:29:20.369139 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:29:20.369123    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5\": container with ID starting with 243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5 not found: ID does not exist" containerID="243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5"
Sep 17 19:29:20.369217 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.369148    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5"} err="failed to get container status \"243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5\": rpc error: code = NotFound desc = could not find container \"243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5\": container with ID starting with 243220dd1c36bcbf8f9d81d3faa59ca234748cab4cadfbebb08315ed158d2fb5 not found: ID does not exist"
Sep 17 19:29:20.369217 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.369164    2583 scope.go:117] "RemoveContainer" containerID="c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd"
Sep 17 19:29:20.369379 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:29:20.369364    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd\": container with ID starting with c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd not found: ID does not exist" containerID="c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd"
Sep 17 19:29:20.369435 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.369382    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd"} err="failed to get container status \"c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd\": rpc error: code = NotFound desc = could not find container \"c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd\": container with ID starting with c34e8cccbfa09b9d691cece8bb4efe3b48b441f59e782dc4cf1436cc5b24befd not found: ID does not exist"
Sep 17 19:29:20.369435 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.369393    2583 scope.go:117] "RemoveContainer" containerID="e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91"
Sep 17 19:29:20.369658 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:29:20.369638    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91\": container with ID starting with e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91 not found: ID does not exist" containerID="e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91"
Sep 17 19:29:20.369731 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.369665    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91"} err="failed to get container status \"e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91\": rpc error: code = NotFound desc = could not find container \"e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91\": container with ID starting with e833453b90b515d1138ecd81a54eebb97772e8f41759446e029cf6ee8cf23c91 not found: ID does not exist"
Sep 17 19:29:20.370218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.370198    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-qtq44"]
Sep 17 19:29:20.374993 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.374977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.375051 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.375004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eabc9730-35e2-4947-b915-c7da39f13f5f-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.375051 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.375047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5c87\" (UniqueName: \"kubernetes.io/projected/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-api-access-w5c87\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.375161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.375067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.375161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.375107    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eabc9730-35e2-4947-b915-c7da39f13f5f-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.375474 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.375456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eabc9730-35e2-4947-b915-c7da39f13f5f-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.375684 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.375667    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.375735 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.375676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.377478 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.377461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eabc9730-35e2-4947-b915-c7da39f13f5f-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.383999 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.383978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5c87\" (UniqueName: \"kubernetes.io/projected/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-api-access-w5c87\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.495174 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.495133    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:29:20.774937 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:20.774914    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"]
Sep 17 19:29:20.777504 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:29:20.777479    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabc9730_35e2_4947_b915_c7da39f13f5f.slice/crio-aaeec2de38c5f68fee362b9c469b474ad5ef92d2762f3ccc2aa00c92ff3c5f29 WatchSource:0}: Error finding container aaeec2de38c5f68fee362b9c469b474ad5ef92d2762f3ccc2aa00c92ff3c5f29: Status 404 returned error can't find the container with id aaeec2de38c5f68fee362b9c469b474ad5ef92d2762f3ccc2aa00c92ff3c5f29
Sep 17 19:29:21.038922 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:21.038842    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9774bac4-dc6d-498f-8a76-edcada37d194" path="/var/lib/kubelet/pods/9774bac4-dc6d-498f-8a76-edcada37d194/volumes"
Sep 17 19:29:21.350490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:21.350404    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" event={"ID":"eabc9730-35e2-4947-b915-c7da39f13f5f","Type":"ContainerStarted","Data":"5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396"}
Sep 17 19:29:21.350490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:21.350471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" event={"ID":"eabc9730-35e2-4947-b915-c7da39f13f5f","Type":"ContainerStarted","Data":"aaeec2de38c5f68fee362b9c469b474ad5ef92d2762f3ccc2aa00c92ff3c5f29"}
Sep 17 19:29:25.368119 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:25.368088    2583 generic.go:356] "Generic (PLEG): container finished" podID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerID="5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396" exitCode=0
Sep 17 19:29:25.368534 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:25.368161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" event={"ID":"eabc9730-35e2-4947-b915-c7da39f13f5f","Type":"ContainerDied","Data":"5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396"}
Sep 17 19:29:59.076349 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:29:59.076315    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:30:10.076895 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:10.076785    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:30:16.591349 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:16.591313    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"]
Sep 17 19:30:16.591876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:16.591669    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kserve-container" containerID="cri-o://a81af3335b994d101d734a9275ac8e9ea744efc8ffb7973ba6b519a8999ffd28" gracePeriod=30
Sep 17 19:30:16.591963 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:16.591889    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kube-rbac-proxy" containerID="cri-o://3d967a7517441e4bd4f3654ad644311e19826b6636f4a50a50a1075a1fea358d" gracePeriod=30
Sep 17 19:30:17.652200 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:17.652162    2583 generic.go:356] "Generic (PLEG): container finished" podID="97808097-7481-479a-9eb4-84c4154ef102" containerID="3d967a7517441e4bd4f3654ad644311e19826b6636f4a50a50a1075a1fea358d" exitCode=2
Sep 17 19:30:17.652648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:17.652243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" event={"ID":"97808097-7481-479a-9eb4-84c4154ef102","Type":"ContainerDied","Data":"3d967a7517441e4bd4f3654ad644311e19826b6636f4a50a50a1075a1fea358d"}
Sep 17 19:30:18.113281 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.113234    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 19:30:18.660582 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.660546    2583 generic.go:356] "Generic (PLEG): container finished" podID="97808097-7481-479a-9eb4-84c4154ef102" containerID="a81af3335b994d101d734a9275ac8e9ea744efc8ffb7973ba6b519a8999ffd28" exitCode=0
Sep 17 19:30:18.660989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.660594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" event={"ID":"97808097-7481-479a-9eb4-84c4154ef102","Type":"ContainerDied","Data":"a81af3335b994d101d734a9275ac8e9ea744efc8ffb7973ba6b519a8999ffd28"}
Sep 17 19:30:18.660989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.660678    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws" event={"ID":"97808097-7481-479a-9eb4-84c4154ef102","Type":"ContainerDied","Data":"0b88ebf04799b6c26b9920b41f561fe13071702b0e12445e68a444d4126dbd19"}
Sep 17 19:30:18.660989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.660708    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b88ebf04799b6c26b9920b41f561fe13071702b0e12445e68a444d4126dbd19"
Sep 17 19:30:18.680714 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.680687    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:30:18.801797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.801715    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d59hq\" (UniqueName: \"kubernetes.io/projected/97808097-7481-479a-9eb4-84c4154ef102-kube-api-access-d59hq\") pod \"97808097-7481-479a-9eb4-84c4154ef102\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") "
Sep 17 19:30:18.801797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.801778    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97808097-7481-479a-9eb4-84c4154ef102-kserve-provision-location\") pod \"97808097-7481-479a-9eb4-84c4154ef102\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") "
Sep 17 19:30:18.802033 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.801858    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-cabundle-cert\") pod \"97808097-7481-479a-9eb4-84c4154ef102\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") "
Sep 17 19:30:18.802033 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.801894    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97808097-7481-479a-9eb4-84c4154ef102-proxy-tls\") pod \"97808097-7481-479a-9eb4-84c4154ef102\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") "
Sep 17 19:30:18.802033 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.801966    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-kube-rbac-proxy-sar-config\") pod \"97808097-7481-479a-9eb4-84c4154ef102\" (UID: \"97808097-7481-479a-9eb4-84c4154ef102\") "
Sep 17 19:30:18.802237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.802205    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "97808097-7481-479a-9eb4-84c4154ef102" (UID: "97808097-7481-479a-9eb4-84c4154ef102"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:18.802549 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.802469    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97808097-7481-479a-9eb4-84c4154ef102" (UID: "97808097-7481-479a-9eb4-84c4154ef102"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:18.804601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.804574    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97808097-7481-479a-9eb4-84c4154ef102-kube-api-access-d59hq" (OuterVolumeSpecName: "kube-api-access-d59hq") pod "97808097-7481-479a-9eb4-84c4154ef102" (UID: "97808097-7481-479a-9eb4-84c4154ef102"). InnerVolumeSpecName "kube-api-access-d59hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:18.804601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.804582    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97808097-7481-479a-9eb4-84c4154ef102-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97808097-7481-479a-9eb4-84c4154ef102" (UID: "97808097-7481-479a-9eb4-84c4154ef102"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:18.807343 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.807315    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97808097-7481-479a-9eb4-84c4154ef102-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97808097-7481-479a-9eb4-84c4154ef102" (UID: "97808097-7481-479a-9eb4-84c4154ef102"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:18.903054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.903020    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97808097-7481-479a-9eb4-84c4154ef102-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:30:18.903054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.903050    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:30:18.903054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.903061    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97808097-7481-479a-9eb4-84c4154ef102-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:30:18.903347 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.903069    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97808097-7481-479a-9eb4-84c4154ef102-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:30:18.903347 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:18.903077    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d59hq\" (UniqueName: \"kubernetes.io/projected/97808097-7481-479a-9eb4-84c4154ef102-kube-api-access-d59hq\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:30:19.665672 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:19.665640    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"
Sep 17 19:30:19.685408 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:19.685380    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"]
Sep 17 19:30:19.695559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:19.695536    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-27fws"]
Sep 17 19:30:21.039445 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.039414    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97808097-7481-479a-9eb4-84c4154ef102" path="/var/lib/kubelet/pods/97808097-7481-479a-9eb4-84c4154ef102/volumes"
Sep 17 19:30:21.878039 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.878002    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"]
Sep 17 19:30:21.878373 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.878359    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kserve-container"
Sep 17 19:30:21.878373 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.878375    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kserve-container"
Sep 17 19:30:21.878514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.878384    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kube-rbac-proxy"
Sep 17 19:30:21.878514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.878390    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kube-rbac-proxy"
Sep 17 19:30:21.878514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.878399    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="storage-initializer"
Sep 17 19:30:21.878514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.878405    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="storage-initializer"
Sep 17 19:30:21.878514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.878480    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kserve-container"
Sep 17 19:30:21.878514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.878491    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="97808097-7481-479a-9eb4-84c4154ef102" containerName="kube-rbac-proxy"
Sep 17 19:30:21.882137 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.882107    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:21.884940 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.884918    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 19:30:21.885825 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.885805    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 19:30:21.901029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:21.900999    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"]
Sep 17 19:30:22.031229 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.031188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.031413 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.031309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.031413 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.031376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b63c-a44a-43d4-90ff-90952196d996-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.031547 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.031415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7df2b63c-a44a-43d4-90ff-90952196d996-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.031547 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.031480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdcwh\" (UniqueName: \"kubernetes.io/projected/7df2b63c-a44a-43d4-90ff-90952196d996-kube-api-access-pdcwh\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.132471 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.132378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.132935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.132552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b63c-a44a-43d4-90ff-90952196d996-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.132935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.132604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7df2b63c-a44a-43d4-90ff-90952196d996-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.132935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.132665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdcwh\" (UniqueName: \"kubernetes.io/projected/7df2b63c-a44a-43d4-90ff-90952196d996-kube-api-access-pdcwh\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.132935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.132729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.133155 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.133089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7df2b63c-a44a-43d4-90ff-90952196d996-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.133218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.133183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.133395 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.133375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.135924 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.135901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b63c-a44a-43d4-90ff-90952196d996-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.142581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.142541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdcwh\" (UniqueName: \"kubernetes.io/projected/7df2b63c-a44a-43d4-90ff-90952196d996-kube-api-access-pdcwh\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.193940 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.193894    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:30:22.476794 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.476727    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"]
Sep 17 19:30:22.479938 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:30:22.479910    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df2b63c_a44a_43d4_90ff_90952196d996.slice/crio-78ded95a9f64c7bc0b6eaab045a30c7d324088cdc17c8617985b35de394760ec WatchSource:0}: Error finding container 78ded95a9f64c7bc0b6eaab045a30c7d324088cdc17c8617985b35de394760ec: Status 404 returned error can't find the container with id 78ded95a9f64c7bc0b6eaab045a30c7d324088cdc17c8617985b35de394760ec
Sep 17 19:30:22.683638 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.683558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" event={"ID":"7df2b63c-a44a-43d4-90ff-90952196d996","Type":"ContainerStarted","Data":"daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274"}
Sep 17 19:30:22.683826 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:22.683668    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" event={"ID":"7df2b63c-a44a-43d4-90ff-90952196d996","Type":"ContainerStarted","Data":"78ded95a9f64c7bc0b6eaab045a30c7d324088cdc17c8617985b35de394760ec"}
Sep 17 19:30:23.106903 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:23.106862    2583 scope.go:117] "RemoveContainer" containerID="9d58894d9454c18d82a7514c15d30e08cbf1a87aa2b95750b2ddeb510bf7527c"
Sep 17 19:30:23.117045 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:30:23.117001    2583 scope.go:117] "RemoveContainer" containerID="cb7454e1dd87f32a1dabec75a166081cd9bd8199f3bd1fe49cb9c9ec37658699"
Sep 17 19:31:27.754368 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:27.754301    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:31:27.754368 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:27.754302    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:31:28.015242 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:28.015166    2583 generic.go:356] "Generic (PLEG): container finished" podID="7df2b63c-a44a-43d4-90ff-90952196d996" containerID="daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274" exitCode=0
Sep 17 19:31:28.015379 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:28.015243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" event={"ID":"7df2b63c-a44a-43d4-90ff-90952196d996","Type":"ContainerDied","Data":"daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274"}
Sep 17 19:31:29.022522 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:29.022486    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" event={"ID":"7df2b63c-a44a-43d4-90ff-90952196d996","Type":"ContainerStarted","Data":"3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce"}
Sep 17 19:31:29.023131 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:29.022566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" event={"ID":"7df2b63c-a44a-43d4-90ff-90952196d996","Type":"ContainerStarted","Data":"ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b"}
Sep 17 19:31:29.023131 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:29.022820    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:31:29.024767 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:29.024743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" event={"ID":"eabc9730-35e2-4947-b915-c7da39f13f5f","Type":"ContainerStarted","Data":"78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7"}
Sep 17 19:31:29.024890 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:29.024785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" event={"ID":"eabc9730-35e2-4947-b915-c7da39f13f5f","Type":"ContainerStarted","Data":"1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347"}
Sep 17 19:31:29.024957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:29.024905    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:31:29.024957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:29.024953    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:31:29.045197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:29.045143    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" podStartSLOduration=58.730086262 podStartE2EDuration="1m8.045126232s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.31503997s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:29.043951733 +0000 UTC m=+1508.171693148" watchObservedRunningTime="2026-09-17 19:31:29.045126232 +0000 UTC m=+1508.172867633"
Sep 17 19:31:29.065340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:29.065282    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" podStartSLOduration=2.1325639929999998 podStartE2EDuration="2m9.065260684s" totalImagesPullingTime="2m2.888713587s" totalInitContainerRuntime="4.043983104s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:29.064201518 +0000 UTC m=+1508.191942920" watchObservedRunningTime="2026-09-17 19:31:29.065260684 +0000 UTC m=+1508.193002087"
Sep 17 19:31:30.035288 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:30.031161    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:31:30.035288 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:30.031408    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:31:31.034844 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:31.034800    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:31:35.038950 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:35.038922    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:31:36.039105 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:36.039068    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:31:36.083049 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:36.039653    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:31:46.040805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:31:46.040773    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:32:04.457055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:04.457021    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"]
Sep 17 19:32:04.457577 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:04.457381    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kserve-container" containerID="cri-o://ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b" gracePeriod=30
Sep 17 19:32:04.457577 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:04.457411    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kube-rbac-proxy" containerID="cri-o://3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce" gracePeriod=30
Sep 17 19:32:05.144694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:05.144637    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 19:32:05.190995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:05.190956    2583 generic.go:356] "Generic (PLEG): container finished" podID="7df2b63c-a44a-43d4-90ff-90952196d996" containerID="3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce" exitCode=2
Sep 17 19:32:05.190995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:05.190987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" event={"ID":"7df2b63c-a44a-43d4-90ff-90952196d996","Type":"ContainerDied","Data":"3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce"}
Sep 17 19:32:06.035032 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.034998    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 19:32:06.040575 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.040546    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:32:06.168556 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.168526    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:32:06.197306 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.197227    2583 generic.go:356] "Generic (PLEG): container finished" podID="7df2b63c-a44a-43d4-90ff-90952196d996" containerID="ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b" exitCode=0
Sep 17 19:32:06.197306 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.197284    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"
Sep 17 19:32:06.197539 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.197304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" event={"ID":"7df2b63c-a44a-43d4-90ff-90952196d996","Type":"ContainerDied","Data":"ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b"}
Sep 17 19:32:06.197539 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.197363    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm" event={"ID":"7df2b63c-a44a-43d4-90ff-90952196d996","Type":"ContainerDied","Data":"78ded95a9f64c7bc0b6eaab045a30c7d324088cdc17c8617985b35de394760ec"}
Sep 17 19:32:06.197539 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.197392    2583 scope.go:117] "RemoveContainer" containerID="3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce"
Sep 17 19:32:06.205849 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.205830    2583 scope.go:117] "RemoveContainer" containerID="ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b"
Sep 17 19:32:06.213362 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.213347    2583 scope.go:117] "RemoveContainer" containerID="daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274"
Sep 17 19:32:06.220758 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.220739    2583 scope.go:117] "RemoveContainer" containerID="3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce"
Sep 17 19:32:06.221036 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:32:06.221019    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce\": container with ID starting with 3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce not found: ID does not exist" containerID="3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce"
Sep 17 19:32:06.221096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.221045    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce"} err="failed to get container status \"3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce\": rpc error: code = NotFound desc = could not find container \"3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce\": container with ID starting with 3d268eadb414a687909ffc624bbb759fdf1f21599bced17c0c43c7953a5832ce not found: ID does not exist"
Sep 17 19:32:06.221096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.221061    2583 scope.go:117] "RemoveContainer" containerID="ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b"
Sep 17 19:32:06.221269 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:32:06.221255    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b\": container with ID starting with ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b not found: ID does not exist" containerID="ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b"
Sep 17 19:32:06.221313 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.221273    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b"} err="failed to get container status \"ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b\": rpc error: code = NotFound desc = could not find container \"ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b\": container with ID starting with ad72307de81e4dd54bcbd2106c49b0253e0b15b2fb76d0b67c5829e210724b3b not found: ID does not exist"
Sep 17 19:32:06.221313 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.221286    2583 scope.go:117] "RemoveContainer" containerID="daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274"
Sep 17 19:32:06.221499 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:32:06.221480    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274\": container with ID starting with daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274 not found: ID does not exist" containerID="daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274"
Sep 17 19:32:06.221540 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.221503    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274"} err="failed to get container status \"daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274\": rpc error: code = NotFound desc = could not find container \"daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274\": container with ID starting with daaaadaf7048f3704ea1a84c0947d61a29b5f33064db51ae2ff0973bd7f74274 not found: ID does not exist"
Sep 17 19:32:06.286976 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.286949    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b63c-a44a-43d4-90ff-90952196d996-proxy-tls\") pod \"7df2b63c-a44a-43d4-90ff-90952196d996\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") "
Sep 17 19:32:06.287107 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.287018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-cabundle-cert\") pod \"7df2b63c-a44a-43d4-90ff-90952196d996\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") "
Sep 17 19:32:06.287107 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.287063    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdcwh\" (UniqueName: \"kubernetes.io/projected/7df2b63c-a44a-43d4-90ff-90952196d996-kube-api-access-pdcwh\") pod \"7df2b63c-a44a-43d4-90ff-90952196d996\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") "
Sep 17 19:32:06.287107 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.287097    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-kube-rbac-proxy-sar-config\") pod \"7df2b63c-a44a-43d4-90ff-90952196d996\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") "
Sep 17 19:32:06.287274 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.287137    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7df2b63c-a44a-43d4-90ff-90952196d996-kserve-provision-location\") pod \"7df2b63c-a44a-43d4-90ff-90952196d996\" (UID: \"7df2b63c-a44a-43d4-90ff-90952196d996\") "
Sep 17 19:32:06.287522 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.287494    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7df2b63c-a44a-43d4-90ff-90952196d996" (UID: "7df2b63c-a44a-43d4-90ff-90952196d996"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:06.287638 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.287496    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7df2b63c-a44a-43d4-90ff-90952196d996" (UID: "7df2b63c-a44a-43d4-90ff-90952196d996"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:06.288218 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.288191    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df2b63c-a44a-43d4-90ff-90952196d996-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7df2b63c-a44a-43d4-90ff-90952196d996" (UID: "7df2b63c-a44a-43d4-90ff-90952196d996"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:06.289169 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.289146    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df2b63c-a44a-43d4-90ff-90952196d996-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7df2b63c-a44a-43d4-90ff-90952196d996" (UID: "7df2b63c-a44a-43d4-90ff-90952196d996"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:32:06.289378 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.289359    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df2b63c-a44a-43d4-90ff-90952196d996-kube-api-access-pdcwh" (OuterVolumeSpecName: "kube-api-access-pdcwh") pod "7df2b63c-a44a-43d4-90ff-90952196d996" (UID: "7df2b63c-a44a-43d4-90ff-90952196d996"). InnerVolumeSpecName "kube-api-access-pdcwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:06.388329 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.388294    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:06.388490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.388345    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdcwh\" (UniqueName: \"kubernetes.io/projected/7df2b63c-a44a-43d4-90ff-90952196d996-kube-api-access-pdcwh\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:06.388490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.388358    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b63c-a44a-43d4-90ff-90952196d996-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:06.388490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.388371    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7df2b63c-a44a-43d4-90ff-90952196d996-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:06.388490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.388384    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b63c-a44a-43d4-90ff-90952196d996-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:06.519176 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.519143    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"]
Sep 17 19:32:06.523123 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:06.523097    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-677zm"]
Sep 17 19:32:07.038784 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:07.038753    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" path="/var/lib/kubelet/pods/7df2b63c-a44a-43d4-90ff-90952196d996/volumes"
Sep 17 19:32:09.719628 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.719580    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"]
Sep 17 19:32:09.720097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.719923    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kserve-container"
Sep 17 19:32:09.720097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.719937    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kserve-container"
Sep 17 19:32:09.720097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.719948    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kube-rbac-proxy"
Sep 17 19:32:09.720097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.719953    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kube-rbac-proxy"
Sep 17 19:32:09.720097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.719971    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="storage-initializer"
Sep 17 19:32:09.720097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.719977    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="storage-initializer"
Sep 17 19:32:09.720097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.720019    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kube-rbac-proxy"
Sep 17 19:32:09.720097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.720027    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df2b63c-a44a-43d4-90ff-90952196d996" containerName="kserve-container"
Sep 17 19:32:09.723381 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.723363    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.726147 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.726126    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 19:32:09.726366 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.726351    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 19:32:09.738988 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.738964    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"]
Sep 17 19:32:09.816144 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.816109    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0e523ca-066c-4385-9157-e24cc6664bca-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.816144 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.816147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.816333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.816210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfpl\" (UniqueName: \"kubernetes.io/projected/b0e523ca-066c-4385-9157-e24cc6664bca-kube-api-access-mmfpl\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.816333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.816266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e523ca-066c-4385-9157-e24cc6664bca-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.816333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.816297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.917004 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.916967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e523ca-066c-4385-9157-e24cc6664bca-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.917152 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.917012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.917213 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.917145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0e523ca-066c-4385-9157-e24cc6664bca-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.917213 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.917182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.917307 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.917214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfpl\" (UniqueName: \"kubernetes.io/projected/b0e523ca-066c-4385-9157-e24cc6664bca-kube-api-access-mmfpl\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.917428 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.917405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e523ca-066c-4385-9157-e24cc6664bca-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.917821 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.917792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.918052 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.918034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.919947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.919928    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0e523ca-066c-4385-9157-e24cc6664bca-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:09.925697 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:09.925678    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfpl\" (UniqueName: \"kubernetes.io/projected/b0e523ca-066c-4385-9157-e24cc6664bca-kube-api-access-mmfpl\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:10.033318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:10.033250    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:10.310200 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:10.310128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"]
Sep 17 19:32:10.313590 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:32:10.313557    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e523ca_066c_4385_9157_e24cc6664bca.slice/crio-2e462712b541cc022c886734c8bda961c1a99c4a5b80853ac42fd4591e857fea WatchSource:0}: Error finding container 2e462712b541cc022c886734c8bda961c1a99c4a5b80853ac42fd4591e857fea: Status 404 returned error can't find the container with id 2e462712b541cc022c886734c8bda961c1a99c4a5b80853ac42fd4591e857fea
Sep 17 19:32:11.221056 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:11.221014    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" event={"ID":"b0e523ca-066c-4385-9157-e24cc6664bca","Type":"ContainerStarted","Data":"e099049d8abea96a899edc24d1b76d083ab552a471d5210ef2f2d6c7ae89375c"}
Sep 17 19:32:11.221442 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:11.221083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" event={"ID":"b0e523ca-066c-4385-9157-e24cc6664bca","Type":"ContainerStarted","Data":"2e462712b541cc022c886734c8bda961c1a99c4a5b80853ac42fd4591e857fea"}
Sep 17 19:32:15.041644 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:15.041590    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:32:15.241854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:15.241760    2583 generic.go:356] "Generic (PLEG): container finished" podID="b0e523ca-066c-4385-9157-e24cc6664bca" containerID="e099049d8abea96a899edc24d1b76d083ab552a471d5210ef2f2d6c7ae89375c" exitCode=0
Sep 17 19:32:15.241854 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:15.241814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" event={"ID":"b0e523ca-066c-4385-9157-e24cc6664bca","Type":"ContainerDied","Data":"e099049d8abea96a899edc24d1b76d083ab552a471d5210ef2f2d6c7ae89375c"}
Sep 17 19:32:16.249755 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:16.249715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" event={"ID":"b0e523ca-066c-4385-9157-e24cc6664bca","Type":"ContainerStarted","Data":"08106ca21f7b9762cfb73af2770b73201fa768b41874eb0e6b272b33dab534bb"}
Sep 17 19:32:16.250194 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:16.249800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" event={"ID":"b0e523ca-066c-4385-9157-e24cc6664bca","Type":"ContainerStarted","Data":"623530bf4963239c111a745f46e31bc0ac767d76c70b60a46944c3587483e2b3"}
Sep 17 19:32:16.250194 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:16.250029    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:16.269082 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:16.269011    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" podStartSLOduration=2.713696905 podStartE2EDuration="7.268997987s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.555301082s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:32:16.267813728 +0000 UTC m=+1555.395555128" watchObservedRunningTime="2026-09-17 19:32:16.268997987 +0000 UTC m=+1555.396739388"
Sep 17 19:32:17.254678 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:17.254640    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:17.255867 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:17.255837    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:32:18.258787 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:18.258745    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:32:20.316683 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:20.316646    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"]
Sep 17 19:32:20.317143 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:20.316968    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kserve-container" containerID="cri-o://1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347" gracePeriod=30
Sep 17 19:32:20.317143 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:20.317057    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kube-rbac-proxy" containerID="cri-o://78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7" gracePeriod=30
Sep 17 19:32:21.276086 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:21.276053    2583 generic.go:356] "Generic (PLEG): container finished" podID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerID="78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7" exitCode=2
Sep 17 19:32:21.276265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:21.276133    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" event={"ID":"eabc9730-35e2-4947-b915-c7da39f13f5f","Type":"ContainerDied","Data":"78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7"}
Sep 17 19:32:23.264289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:23.264257    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:23.264892 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:23.264862    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:32:25.030911 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.030856    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 19:32:25.039805 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.039770    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 19:32:25.491161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.491117    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"]
Sep 17 19:32:25.494671 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.494648    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.497003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.496980    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 19:32:25.497190 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.497162    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:32:25.505051 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.505030    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"]
Sep 17 19:32:25.655964 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.655923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkv6\" (UniqueName: \"kubernetes.io/projected/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-api-access-mxkv6\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.655964 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.655963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.656225 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.656025    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.656225 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.656073    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.656225 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.656106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.756810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.756713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkv6\" (UniqueName: \"kubernetes.io/projected/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-api-access-mxkv6\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.756810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.756756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.756810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.756789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.756810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.756816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.757281 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.756919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.757281 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.757165    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.757565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.757541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.757700 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.757541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.759315 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.759291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.766006 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.765983    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkv6\" (UniqueName: \"kubernetes.io/projected/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-api-access-mxkv6\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:25.805486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:25.805444    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:26.078461 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:26.078321    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"]
Sep 17 19:32:26.081031 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:32:26.081005    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e96ced7_48f5_4ab2_8b50_e24cb200e78c.slice/crio-d4fe0952b504c1efc975dd5ef233d53f24cfd96817f2aa0226e489a5a8a8deb9 WatchSource:0}: Error finding container d4fe0952b504c1efc975dd5ef233d53f24cfd96817f2aa0226e489a5a8a8deb9: Status 404 returned error can't find the container with id d4fe0952b504c1efc975dd5ef233d53f24cfd96817f2aa0226e489a5a8a8deb9
Sep 17 19:32:26.301318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:26.301275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" event={"ID":"6e96ced7-48f5-4ab2-8b50-e24cb200e78c","Type":"ContainerStarted","Data":"c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57"}
Sep 17 19:32:26.301494 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:26.301352    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" event={"ID":"6e96ced7-48f5-4ab2-8b50-e24cb200e78c","Type":"ContainerStarted","Data":"d4fe0952b504c1efc975dd5ef233d53f24cfd96817f2aa0226e489a5a8a8deb9"}
Sep 17 19:32:27.564870 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.564840    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:32:27.673735 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.673711    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eabc9730-35e2-4947-b915-c7da39f13f5f-proxy-tls\") pod \"eabc9730-35e2-4947-b915-c7da39f13f5f\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") "
Sep 17 19:32:27.673876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.673771    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-rbac-proxy-sar-config\") pod \"eabc9730-35e2-4947-b915-c7da39f13f5f\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") "
Sep 17 19:32:27.673876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.673820    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eabc9730-35e2-4947-b915-c7da39f13f5f-kserve-provision-location\") pod \"eabc9730-35e2-4947-b915-c7da39f13f5f\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") "
Sep 17 19:32:27.673876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.673842    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5c87\" (UniqueName: \"kubernetes.io/projected/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-api-access-w5c87\") pod \"eabc9730-35e2-4947-b915-c7da39f13f5f\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") "
Sep 17 19:32:27.673876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.673862    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-cabundle-cert\") pod \"eabc9730-35e2-4947-b915-c7da39f13f5f\" (UID: \"eabc9730-35e2-4947-b915-c7da39f13f5f\") "
Sep 17 19:32:27.674203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.674170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eabc9730-35e2-4947-b915-c7da39f13f5f" (UID: "eabc9730-35e2-4947-b915-c7da39f13f5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:27.674203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.674176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabc9730-35e2-4947-b915-c7da39f13f5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eabc9730-35e2-4947-b915-c7da39f13f5f" (UID: "eabc9730-35e2-4947-b915-c7da39f13f5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:27.674358 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.674264    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eabc9730-35e2-4947-b915-c7da39f13f5f" (UID: "eabc9730-35e2-4947-b915-c7da39f13f5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:27.676068 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.676047    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-api-access-w5c87" (OuterVolumeSpecName: "kube-api-access-w5c87") pod "eabc9730-35e2-4947-b915-c7da39f13f5f" (UID: "eabc9730-35e2-4947-b915-c7da39f13f5f"). InnerVolumeSpecName "kube-api-access-w5c87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:27.676144 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.676100    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabc9730-35e2-4947-b915-c7da39f13f5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eabc9730-35e2-4947-b915-c7da39f13f5f" (UID: "eabc9730-35e2-4947-b915-c7da39f13f5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:32:27.774431 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.774401    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eabc9730-35e2-4947-b915-c7da39f13f5f-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:27.774431 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.774426    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5c87\" (UniqueName: \"kubernetes.io/projected/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-api-access-w5c87\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:27.774594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.774439    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:27.774594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.774452    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eabc9730-35e2-4947-b915-c7da39f13f5f-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:27.774594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:27.774463    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eabc9730-35e2-4947-b915-c7da39f13f5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:28.311442 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.311407    2583 generic.go:356] "Generic (PLEG): container finished" podID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerID="1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347" exitCode=0
Sep 17 19:32:28.311628 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.311470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" event={"ID":"eabc9730-35e2-4947-b915-c7da39f13f5f","Type":"ContainerDied","Data":"1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347"}
Sep 17 19:32:28.311628 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.311472    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"
Sep 17 19:32:28.311628 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.311518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj" event={"ID":"eabc9730-35e2-4947-b915-c7da39f13f5f","Type":"ContainerDied","Data":"aaeec2de38c5f68fee362b9c469b474ad5ef92d2762f3ccc2aa00c92ff3c5f29"}
Sep 17 19:32:28.311628 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.311545    2583 scope.go:117] "RemoveContainer" containerID="78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7"
Sep 17 19:32:28.321082 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.321065    2583 scope.go:117] "RemoveContainer" containerID="1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347"
Sep 17 19:32:28.328726 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.328706    2583 scope.go:117] "RemoveContainer" containerID="5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396"
Sep 17 19:32:28.332898 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.332874    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"]
Sep 17 19:32:28.336841 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.336826    2583 scope.go:117] "RemoveContainer" containerID="78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7"
Sep 17 19:32:28.337106 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:32:28.337085    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7\": container with ID starting with 78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7 not found: ID does not exist" containerID="78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7"
Sep 17 19:32:28.337153 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.337116    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7"} err="failed to get container status \"78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7\": rpc error: code = NotFound desc = could not find container \"78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7\": container with ID starting with 78bbc73fb45bd5bf71637133ad10cb6db4fc6e9476baa9a02f442b865828e2a7 not found: ID does not exist"
Sep 17 19:32:28.337153 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.337131    2583 scope.go:117] "RemoveContainer" containerID="1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347"
Sep 17 19:32:28.337356 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:32:28.337341    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347\": container with ID starting with 1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347 not found: ID does not exist" containerID="1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347"
Sep 17 19:32:28.337398 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.337360    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347"} err="failed to get container status \"1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347\": rpc error: code = NotFound desc = could not find container \"1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347\": container with ID starting with 1816ca256f0040ea840ca8c931007d40e774e693b1a4dc40db99e286804d5347 not found: ID does not exist"
Sep 17 19:32:28.337398 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.337373    2583 scope.go:117] "RemoveContainer" containerID="5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396"
Sep 17 19:32:28.337545 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.337529    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdwfj"]
Sep 17 19:32:28.337588 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:32:28.337566    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396\": container with ID starting with 5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396 not found: ID does not exist" containerID="5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396"
Sep 17 19:32:28.337641 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:28.337590    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396"} err="failed to get container status \"5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396\": rpc error: code = NotFound desc = could not find container \"5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396\": container with ID starting with 5cbf0ebe1c9995ea8a6e1b74a0d2b02e0e5a5a1e03c5f86e8e15f3d32786c396 not found: ID does not exist"
Sep 17 19:32:29.038563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:29.038530    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" path="/var/lib/kubelet/pods/eabc9730-35e2-4947-b915-c7da39f13f5f/volumes"
Sep 17 19:32:32.335019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:32.334986    2583 generic.go:356] "Generic (PLEG): container finished" podID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerID="c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57" exitCode=0
Sep 17 19:32:32.335393 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:32.335060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" event={"ID":"6e96ced7-48f5-4ab2-8b50-e24cb200e78c","Type":"ContainerDied","Data":"c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57"}
Sep 17 19:32:33.265784 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:33.265756    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:33.341863 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:33.341831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" event={"ID":"6e96ced7-48f5-4ab2-8b50-e24cb200e78c","Type":"ContainerStarted","Data":"c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22"}
Sep 17 19:32:33.342287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:33.341893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" event={"ID":"6e96ced7-48f5-4ab2-8b50-e24cb200e78c","Type":"ContainerStarted","Data":"ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f"}
Sep 17 19:32:33.342287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:33.342209    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:33.342383 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:33.342346    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:33.343533 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:33.343507    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:32:33.362513 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:33.362467    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" podStartSLOduration=2.233478659 podStartE2EDuration="8.3624527s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.128974041s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:32:33.360913789 +0000 UTC m=+1572.488655201" watchObservedRunningTime="2026-09-17 19:32:33.3624527 +0000 UTC m=+1572.490194101"
Sep 17 19:32:34.346665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:34.346610    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:32:39.350877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:39.350852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:39.351376 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:39.351347    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:32:45.040498 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:45.040469    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"]
Sep 17 19:32:45.041007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:45.040730    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kserve-container" containerID="cri-o://623530bf4963239c111a745f46e31bc0ac767d76c70b60a46944c3587483e2b3" gracePeriod=30
Sep 17 19:32:45.041007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:45.040750    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kube-rbac-proxy" containerID="cri-o://08106ca21f7b9762cfb73af2770b73201fa768b41874eb0e6b272b33dab534bb" gracePeriod=30
Sep 17 19:32:45.402997 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:45.402910    2583 generic.go:356] "Generic (PLEG): container finished" podID="b0e523ca-066c-4385-9157-e24cc6664bca" containerID="08106ca21f7b9762cfb73af2770b73201fa768b41874eb0e6b272b33dab534bb" exitCode=2
Sep 17 19:32:45.403160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:45.402986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" event={"ID":"b0e523ca-066c-4385-9157-e24cc6664bca","Type":"ContainerDied","Data":"08106ca21f7b9762cfb73af2770b73201fa768b41874eb0e6b272b33dab534bb"}
Sep 17 19:32:46.411677 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.411642    2583 generic.go:356] "Generic (PLEG): container finished" podID="b0e523ca-066c-4385-9157-e24cc6664bca" containerID="623530bf4963239c111a745f46e31bc0ac767d76c70b60a46944c3587483e2b3" exitCode=0
Sep 17 19:32:46.411677 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.411654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" event={"ID":"b0e523ca-066c-4385-9157-e24cc6664bca","Type":"ContainerDied","Data":"623530bf4963239c111a745f46e31bc0ac767d76c70b60a46944c3587483e2b3"}
Sep 17 19:32:46.485017 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.484994    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:46.543828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.543777    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e523ca-066c-4385-9157-e24cc6664bca-kserve-provision-location\") pod \"b0e523ca-066c-4385-9157-e24cc6664bca\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") "
Sep 17 19:32:46.543828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.543812    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-cabundle-cert\") pod \"b0e523ca-066c-4385-9157-e24cc6664bca\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") "
Sep 17 19:32:46.544085 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.543861    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-kube-rbac-proxy-sar-config\") pod \"b0e523ca-066c-4385-9157-e24cc6664bca\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") "
Sep 17 19:32:46.544085 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.543904    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0e523ca-066c-4385-9157-e24cc6664bca-proxy-tls\") pod \"b0e523ca-066c-4385-9157-e24cc6664bca\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") "
Sep 17 19:32:46.544085 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.543924    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmfpl\" (UniqueName: \"kubernetes.io/projected/b0e523ca-066c-4385-9157-e24cc6664bca-kube-api-access-mmfpl\") pod \"b0e523ca-066c-4385-9157-e24cc6664bca\" (UID: \"b0e523ca-066c-4385-9157-e24cc6664bca\") "
Sep 17 19:32:46.544256 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.544233    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0e523ca-066c-4385-9157-e24cc6664bca" (UID: "b0e523ca-066c-4385-9157-e24cc6664bca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:46.544320 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.544247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0e523ca-066c-4385-9157-e24cc6664bca" (UID: "b0e523ca-066c-4385-9157-e24cc6664bca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:46.544791 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.544730    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e523ca-066c-4385-9157-e24cc6664bca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0e523ca-066c-4385-9157-e24cc6664bca" (UID: "b0e523ca-066c-4385-9157-e24cc6664bca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:46.546149 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.546115    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e523ca-066c-4385-9157-e24cc6664bca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0e523ca-066c-4385-9157-e24cc6664bca" (UID: "b0e523ca-066c-4385-9157-e24cc6664bca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:32:46.546275 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.546192    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e523ca-066c-4385-9157-e24cc6664bca-kube-api-access-mmfpl" (OuterVolumeSpecName: "kube-api-access-mmfpl") pod "b0e523ca-066c-4385-9157-e24cc6664bca" (UID: "b0e523ca-066c-4385-9157-e24cc6664bca"). InnerVolumeSpecName "kube-api-access-mmfpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:46.645116 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.645078    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:46.645116 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.645107    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0e523ca-066c-4385-9157-e24cc6664bca-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:46.645116 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.645118    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmfpl\" (UniqueName: \"kubernetes.io/projected/b0e523ca-066c-4385-9157-e24cc6664bca-kube-api-access-mmfpl\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:46.645116 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.645126    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e523ca-066c-4385-9157-e24cc6664bca-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:46.645477 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:46.645136    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0e523ca-066c-4385-9157-e24cc6664bca-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:47.418465 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:47.418431    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg" event={"ID":"b0e523ca-066c-4385-9157-e24cc6664bca","Type":"ContainerDied","Data":"2e462712b541cc022c886734c8bda961c1a99c4a5b80853ac42fd4591e857fea"}
Sep 17 19:32:47.418938 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:47.418488    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"
Sep 17 19:32:47.418938 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:47.418518    2583 scope.go:117] "RemoveContainer" containerID="08106ca21f7b9762cfb73af2770b73201fa768b41874eb0e6b272b33dab534bb"
Sep 17 19:32:47.427267 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:47.427249    2583 scope.go:117] "RemoveContainer" containerID="623530bf4963239c111a745f46e31bc0ac767d76c70b60a46944c3587483e2b3"
Sep 17 19:32:47.437480 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:47.437460    2583 scope.go:117] "RemoveContainer" containerID="e099049d8abea96a899edc24d1b76d083ab552a471d5210ef2f2d6c7ae89375c"
Sep 17 19:32:47.438096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:47.438071    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"]
Sep 17 19:32:47.442580 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:47.442554    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h7dmg"]
Sep 17 19:32:49.038702 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:49.038662    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" path="/var/lib/kubelet/pods/b0e523ca-066c-4385-9157-e24cc6664bca/volumes"
Sep 17 19:32:49.351734 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:49.351653    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:50.259933 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.259898    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"]
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260239    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kserve-container"
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260252    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kserve-container"
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260265    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="storage-initializer"
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260270    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="storage-initializer"
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260289    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kube-rbac-proxy"
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260296    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kube-rbac-proxy"
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260307    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="storage-initializer"
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260311    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="storage-initializer"
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260320    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kube-rbac-proxy"
Sep 17 19:32:50.260330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260328    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kube-rbac-proxy"
Sep 17 19:32:50.260705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260339    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kserve-container"
Sep 17 19:32:50.260705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260347    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kserve-container"
Sep 17 19:32:50.260705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260399    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kube-rbac-proxy"
Sep 17 19:32:50.260705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260415    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kube-rbac-proxy"
Sep 17 19:32:50.260705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260424    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e523ca-066c-4385-9157-e24cc6664bca" containerName="kserve-container"
Sep 17 19:32:50.260705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.260431    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eabc9730-35e2-4947-b915-c7da39f13f5f" containerName="kserve-container"
Sep 17 19:32:50.276281 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.276251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"]
Sep 17 19:32:50.276466 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.276356    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.281597 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.281573    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 19:32:50.281597 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.281595    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 19:32:50.357482 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.357434    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"]
Sep 17 19:32:50.360707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.360683    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.363102 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.363078    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 19:32:50.376370 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.376338    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"]
Sep 17 19:32:50.377129 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.377103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4177a21-86ae-4281-b6b3-eae6117395ba-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.377229 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.377198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4177a21-86ae-4281-b6b3-eae6117395ba-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.377289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.377231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.377289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.377258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqcbf\" (UniqueName: \"kubernetes.io/projected/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-api-access-cqcbf\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.377437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.377305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.478119 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4177a21-86ae-4281-b6b3-eae6117395ba-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.478119 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.478382 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqcbf\" (UniqueName: \"kubernetes.io/projected/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-api-access-cqcbf\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.478382 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw46r\" (UniqueName: \"kubernetes.io/projected/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-api-access-bw46r\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.478382 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.478537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec35817e-67ad-48cc-8836-4df13a6cb298-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.478537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.478537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.478707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478569    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4177a21-86ae-4281-b6b3-eae6117395ba-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.478707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec35817e-67ad-48cc-8836-4df13a6cb298-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.478951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.478930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4177a21-86ae-4281-b6b3-eae6117395ba-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.479028 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.479007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.479067 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.479007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.480740 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.480717    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4177a21-86ae-4281-b6b3-eae6117395ba-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.487710 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.487687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqcbf\" (UniqueName: \"kubernetes.io/projected/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-api-access-cqcbf\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.579369 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.579300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec35817e-67ad-48cc-8836-4df13a6cb298-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.579369 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.579338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.579369 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.579360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.579566 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.579404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec35817e-67ad-48cc-8836-4df13a6cb298-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.579566 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.579471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw46r\" (UniqueName: \"kubernetes.io/projected/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-api-access-bw46r\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.579917 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.579893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec35817e-67ad-48cc-8836-4df13a6cb298-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.580088 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.580043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.580088 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.580050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.581816 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.581799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec35817e-67ad-48cc-8836-4df13a6cb298-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.586775 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.586755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:50.589682 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.589482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw46r\" (UniqueName: \"kubernetes.io/projected/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-api-access-bw46r\") pod \"isvc-canary-create-predictor-7c7f45c769-zjdl7\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.671377 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.671336    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:50.859492 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.859469    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"]
Sep 17 19:32:50.861820 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:32:50.861793    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4177a21_86ae_4281_b6b3_eae6117395ba.slice/crio-80385ccedfb15a1bc54d671825993e7b5536f529110fd4978955f5b26f3c7e90 WatchSource:0}: Error finding container 80385ccedfb15a1bc54d671825993e7b5536f529110fd4978955f5b26f3c7e90: Status 404 returned error can't find the container with id 80385ccedfb15a1bc54d671825993e7b5536f529110fd4978955f5b26f3c7e90
Sep 17 19:32:50.947766 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:50.947742    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"]
Sep 17 19:32:50.950095 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:32:50.950062    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec35817e_67ad_48cc_8836_4df13a6cb298.slice/crio-9fac67c6cf61a1cceb12ea2145b5ea6f73dae95444f5a43aa81310293eff80a8 WatchSource:0}: Error finding container 9fac67c6cf61a1cceb12ea2145b5ea6f73dae95444f5a43aa81310293eff80a8: Status 404 returned error can't find the container with id 9fac67c6cf61a1cceb12ea2145b5ea6f73dae95444f5a43aa81310293eff80a8
Sep 17 19:32:51.443512 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:51.443473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" event={"ID":"e4177a21-86ae-4281-b6b3-eae6117395ba","Type":"ContainerStarted","Data":"de8a33a1b3e45543f9311bbc2befb90cc8a597f0a5d6327d2cff7085bd24794e"}
Sep 17 19:32:51.443989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:51.443544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" event={"ID":"e4177a21-86ae-4281-b6b3-eae6117395ba","Type":"ContainerStarted","Data":"80385ccedfb15a1bc54d671825993e7b5536f529110fd4978955f5b26f3c7e90"}
Sep 17 19:32:51.445397 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:51.445359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" event={"ID":"ec35817e-67ad-48cc-8836-4df13a6cb298","Type":"ContainerStarted","Data":"3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446"}
Sep 17 19:32:51.445512 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:51.445420    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" event={"ID":"ec35817e-67ad-48cc-8836-4df13a6cb298","Type":"ContainerStarted","Data":"9fac67c6cf61a1cceb12ea2145b5ea6f73dae95444f5a43aa81310293eff80a8"}
Sep 17 19:32:55.467323 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:55.467278    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerID="de8a33a1b3e45543f9311bbc2befb90cc8a597f0a5d6327d2cff7085bd24794e" exitCode=0
Sep 17 19:32:55.467752 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:55.467349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" event={"ID":"e4177a21-86ae-4281-b6b3-eae6117395ba","Type":"ContainerDied","Data":"de8a33a1b3e45543f9311bbc2befb90cc8a597f0a5d6327d2cff7085bd24794e"}
Sep 17 19:32:55.469251 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:55.469231    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerID="3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446" exitCode=0
Sep 17 19:32:55.469363 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:55.469292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" event={"ID":"ec35817e-67ad-48cc-8836-4df13a6cb298","Type":"ContainerDied","Data":"3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446"}
Sep 17 19:32:56.479210 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.479162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" event={"ID":"ec35817e-67ad-48cc-8836-4df13a6cb298","Type":"ContainerStarted","Data":"a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b"}
Sep 17 19:32:56.479677 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.479242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" event={"ID":"ec35817e-67ad-48cc-8836-4df13a6cb298","Type":"ContainerStarted","Data":"2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd"}
Sep 17 19:32:56.479677 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.479570    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:56.481866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.481835    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" event={"ID":"e4177a21-86ae-4281-b6b3-eae6117395ba","Type":"ContainerStarted","Data":"9662b3c9652460a298297e8a6146c595fb194b6280d2d30d0c6d04ffa2ed8cc8"}
Sep 17 19:32:56.482008 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.481905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" event={"ID":"e4177a21-86ae-4281-b6b3-eae6117395ba","Type":"ContainerStarted","Data":"5cf33029e0971b197b55075be454d4970893d0d352bc5d70a3613ca6c55aefcb"}
Sep 17 19:32:56.482148 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.482124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:56.482261 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.482178    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:32:56.483539 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.483505    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:32:56.499996 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.499928    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podStartSLOduration=2.673838969 podStartE2EDuration="6.499904536s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.826065567s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:32:56.498633917 +0000 UTC m=+1595.626375314" watchObservedRunningTime="2026-09-17 19:32:56.499904536 +0000 UTC m=+1595.627645938"
Sep 17 19:32:56.517172 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:56.517109    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podStartSLOduration=2.402438256 podStartE2EDuration="6.517091083s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.114652827s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:32:56.515076207 +0000 UTC m=+1595.642817608" watchObservedRunningTime="2026-09-17 19:32:56.517091083 +0000 UTC m=+1595.644832484"
Sep 17 19:32:57.487386 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:57.487349    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:32:57.487818 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:57.487391    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:32:57.488649 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:57.488600    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:32:57.730771 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:57.730743    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"]
Sep 17 19:32:57.731012 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:57.730992    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kserve-container" containerID="cri-o://ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f" gracePeriod=30
Sep 17 19:32:57.731142 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:57.731087    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kube-rbac-proxy" containerID="cri-o://c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22" gracePeriod=30
Sep 17 19:32:58.494152 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:58.494113    2583 generic.go:356] "Generic (PLEG): container finished" podID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerID="c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22" exitCode=2
Sep 17 19:32:58.494509 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:58.494197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" event={"ID":"6e96ced7-48f5-4ab2-8b50-e24cb200e78c","Type":"ContainerDied","Data":"c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22"}
Sep 17 19:32:58.494598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:58.494526    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:32:59.071915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.071893    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:59.155740 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.155673    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxkv6\" (UniqueName: \"kubernetes.io/projected/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-api-access-mxkv6\") pod \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") "
Sep 17 19:32:59.155740 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.155717    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-cabundle-cert\") pod \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") "
Sep 17 19:32:59.155921 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.155751    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kserve-provision-location\") pod \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") "
Sep 17 19:32:59.155921 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.155789    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-rbac-proxy-sar-config\") pod \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") "
Sep 17 19:32:59.156044 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.155957    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-proxy-tls\") pod \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\" (UID: \"6e96ced7-48f5-4ab2-8b50-e24cb200e78c\") "
Sep 17 19:32:59.156088 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.156037    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e96ced7-48f5-4ab2-8b50-e24cb200e78c" (UID: "6e96ced7-48f5-4ab2-8b50-e24cb200e78c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:59.156145 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.156126    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e96ced7-48f5-4ab2-8b50-e24cb200e78c" (UID: "6e96ced7-48f5-4ab2-8b50-e24cb200e78c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:59.156458 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.156435    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:59.156570 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.156462    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:59.158090 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.158068    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e96ced7-48f5-4ab2-8b50-e24cb200e78c" (UID: "6e96ced7-48f5-4ab2-8b50-e24cb200e78c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:32:59.158271 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.158249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-api-access-mxkv6" (OuterVolumeSpecName: "kube-api-access-mxkv6") pod "6e96ced7-48f5-4ab2-8b50-e24cb200e78c" (UID: "6e96ced7-48f5-4ab2-8b50-e24cb200e78c"). InnerVolumeSpecName "kube-api-access-mxkv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:59.182597 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.182570    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e96ced7-48f5-4ab2-8b50-e24cb200e78c" (UID: "6e96ced7-48f5-4ab2-8b50-e24cb200e78c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:59.257484 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.257451    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:59.257484 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.257476    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxkv6\" (UniqueName: \"kubernetes.io/projected/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kube-api-access-mxkv6\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:59.257484 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.257485    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96ced7-48f5-4ab2-8b50-e24cb200e78c-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:32:59.500452 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.500421    2583 generic.go:356] "Generic (PLEG): container finished" podID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerID="ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f" exitCode=0
Sep 17 19:32:59.500935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.500479    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"
Sep 17 19:32:59.500935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.500502    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" event={"ID":"6e96ced7-48f5-4ab2-8b50-e24cb200e78c","Type":"ContainerDied","Data":"ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f"}
Sep 17 19:32:59.500935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.500577    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72" event={"ID":"6e96ced7-48f5-4ab2-8b50-e24cb200e78c","Type":"ContainerDied","Data":"d4fe0952b504c1efc975dd5ef233d53f24cfd96817f2aa0226e489a5a8a8deb9"}
Sep 17 19:32:59.500935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.500642    2583 scope.go:117] "RemoveContainer" containerID="c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22"
Sep 17 19:32:59.509769 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.509752    2583 scope.go:117] "RemoveContainer" containerID="ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f"
Sep 17 19:32:59.517408 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.517390    2583 scope.go:117] "RemoveContainer" containerID="c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57"
Sep 17 19:32:59.523602 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.523564    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"]
Sep 17 19:32:59.526508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.526486    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-tsp72"]
Sep 17 19:32:59.527529 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.527508    2583 scope.go:117] "RemoveContainer" containerID="c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22"
Sep 17 19:32:59.527838 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:32:59.527820    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22\": container with ID starting with c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22 not found: ID does not exist" containerID="c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22"
Sep 17 19:32:59.527894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.527847    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22"} err="failed to get container status \"c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22\": rpc error: code = NotFound desc = could not find container \"c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22\": container with ID starting with c7b5602184feb318b62f99e007e2f3ad5b656ac44d0693823c943808ba11cf22 not found: ID does not exist"
Sep 17 19:32:59.527894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.527871    2583 scope.go:117] "RemoveContainer" containerID="ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f"
Sep 17 19:32:59.528129 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:32:59.528109    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f\": container with ID starting with ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f not found: ID does not exist" containerID="ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f"
Sep 17 19:32:59.528171 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.528132    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f"} err="failed to get container status \"ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f\": rpc error: code = NotFound desc = could not find container \"ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f\": container with ID starting with ccb4cee5f8bee034f5bfb9932d7ea0defdc445b8a4f4ab5d458b08a13c88a21f not found: ID does not exist"
Sep 17 19:32:59.528171 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.528144    2583 scope.go:117] "RemoveContainer" containerID="c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57"
Sep 17 19:32:59.528347 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:32:59.528329    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57\": container with ID starting with c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57 not found: ID does not exist" containerID="c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57"
Sep 17 19:32:59.528384 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:32:59.528353    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57"} err="failed to get container status \"c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57\": rpc error: code = NotFound desc = could not find container \"c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57\": container with ID starting with c132f85b7831865aa091495e7e7a2129a30e0528dc999b542c72ee37f8e2ac57 not found: ID does not exist"
Sep 17 19:33:01.038451 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:01.038419    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" path="/var/lib/kubelet/pods/6e96ced7-48f5-4ab2-8b50-e24cb200e78c/volumes"
Sep 17 19:33:02.492327 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.492259    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:33:02.492787 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.492762    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:33:02.910497 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.910420    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"]
Sep 17 19:33:02.910777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.910764    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kube-rbac-proxy"
Sep 17 19:33:02.910834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.910779    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kube-rbac-proxy"
Sep 17 19:33:02.910834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.910801    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="storage-initializer"
Sep 17 19:33:02.910834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.910806    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="storage-initializer"
Sep 17 19:33:02.910834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.910815    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kserve-container"
Sep 17 19:33:02.910834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.910820    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kserve-container"
Sep 17 19:33:02.910983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.910884    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kserve-container"
Sep 17 19:33:02.910983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.910896    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e96ced7-48f5-4ab2-8b50-e24cb200e78c" containerName="kube-rbac-proxy"
Sep 17 19:33:02.915634 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.915599    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:02.917984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.917959    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:33:02.918106 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.917983    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 19:33:02.923963 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.923944    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"]
Sep 17 19:33:02.990386 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.990352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xlr\" (UniqueName: \"kubernetes.io/projected/826c9186-bbea-479d-a62c-968049a9651f-kube-api-access-d6xlr\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:02.990386 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.990390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826c9186-bbea-479d-a62c-968049a9651f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:02.990578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.990419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:02.990578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.990492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:02.990578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:02.990518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826c9186-bbea-479d-a62c-968049a9651f-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.091929 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.091895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xlr\" (UniqueName: \"kubernetes.io/projected/826c9186-bbea-479d-a62c-968049a9651f-kube-api-access-d6xlr\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.092070 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.091939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826c9186-bbea-479d-a62c-968049a9651f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.092070 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.091984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.092070 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.092030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.092070 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.092058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826c9186-bbea-479d-a62c-968049a9651f-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.092245 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:33:03.092116    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 19:33:03.092245 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:33:03.092180    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/826c9186-bbea-479d-a62c-968049a9651f-proxy-tls podName:826c9186-bbea-479d-a62c-968049a9651f nodeName:}" failed. No retries permitted until 2026-09-17 19:33:03.592159662 +0000 UTC m=+1602.719901040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/826c9186-bbea-479d-a62c-968049a9651f-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" (UID: "826c9186-bbea-479d-a62c-968049a9651f") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 19:33:03.092512 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.092495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826c9186-bbea-479d-a62c-968049a9651f-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.092731 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.092716    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.092731 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.092724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.102901 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.102878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xlr\" (UniqueName: \"kubernetes.io/projected/826c9186-bbea-479d-a62c-968049a9651f-kube-api-access-d6xlr\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.499158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.499126    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:33:03.499760 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.499729    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:33:03.596472 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.596438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826c9186-bbea-479d-a62c-968049a9651f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.599036 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.599008    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826c9186-bbea-479d-a62c-968049a9651f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:03.826576 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:03.826487    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:04.104980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:04.104906    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"]
Sep 17 19:33:04.107258 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:33:04.107229    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826c9186_bbea_479d_a62c_968049a9651f.slice/crio-be1ce5b782cdba5ac34bb2db8f6ff84f429fc7d8729acfd448dd095a19723103 WatchSource:0}: Error finding container be1ce5b782cdba5ac34bb2db8f6ff84f429fc7d8729acfd448dd095a19723103: Status 404 returned error can't find the container with id be1ce5b782cdba5ac34bb2db8f6ff84f429fc7d8729acfd448dd095a19723103
Sep 17 19:33:04.525919 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:04.525885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" event={"ID":"826c9186-bbea-479d-a62c-968049a9651f","Type":"ContainerStarted","Data":"39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d"}
Sep 17 19:33:04.526296 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:04.525947    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" event={"ID":"826c9186-bbea-479d-a62c-968049a9651f","Type":"ContainerStarted","Data":"be1ce5b782cdba5ac34bb2db8f6ff84f429fc7d8729acfd448dd095a19723103"}
Sep 17 19:33:08.547318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:08.547284    2583 generic.go:356] "Generic (PLEG): container finished" podID="826c9186-bbea-479d-a62c-968049a9651f" containerID="39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d" exitCode=0
Sep 17 19:33:08.547806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:08.547347    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" event={"ID":"826c9186-bbea-479d-a62c-968049a9651f","Type":"ContainerDied","Data":"39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d"}
Sep 17 19:33:09.554904 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:09.554865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" event={"ID":"826c9186-bbea-479d-a62c-968049a9651f","Type":"ContainerStarted","Data":"f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0"}
Sep 17 19:33:09.555339 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:09.554929    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" event={"ID":"826c9186-bbea-479d-a62c-968049a9651f","Type":"ContainerStarted","Data":"dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3"}
Sep 17 19:33:09.555339 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:09.555147    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:09.555339 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:09.555326    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:09.579705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:09.579654    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" podStartSLOduration=3.733217304 podStartE2EDuration="7.579638874s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.84642157s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:33:09.578078685 +0000 UTC m=+1608.705820087" watchObservedRunningTime="2026-09-17 19:33:09.579638874 +0000 UTC m=+1608.707380265"
Sep 17 19:33:12.492980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:12.492924    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:33:13.500001 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:13.499962    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:33:15.565725 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:15.565692    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:33:22.493073 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:22.493035    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:33:23.500424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:23.500381    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:33:32.492859 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:32.492815    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:33:33.500382 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:33.500339    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:33:42.493746 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:42.493705    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:33:43.500586 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:43.500550    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:33:45.644662 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:45.644581    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 19:33:52.493276 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:52.493235    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:33:53.500333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:53.500297    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:33:55.569078 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:33:55.569048    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:34:02.493321 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:02.493281    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:34:03.014826 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:03.014793    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"]
Sep 17 19:34:03.015061 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:03.015018    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kserve-container" containerID="cri-o://dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3" gracePeriod=30
Sep 17 19:34:03.015192 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:03.015091    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kube-rbac-proxy" containerID="cri-o://f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0" gracePeriod=30
Sep 17 19:34:03.500447 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:03.500410    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:34:03.799742 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:03.799656    2583 generic.go:356] "Generic (PLEG): container finished" podID="826c9186-bbea-479d-a62c-968049a9651f" containerID="f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0" exitCode=2
Sep 17 19:34:03.799878 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:03.799733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" event={"ID":"826c9186-bbea-479d-a62c-968049a9651f","Type":"ContainerDied","Data":"f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0"}
Sep 17 19:34:04.034267 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:04.034222    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:34:05.561097 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:05.561053    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 19:34:10.034114 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.034072    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:34:10.569877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.569853    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:34:10.666551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.666454    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826c9186-bbea-479d-a62c-968049a9651f-kserve-provision-location\") pod \"826c9186-bbea-479d-a62c-968049a9651f\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") "
Sep 17 19:34:10.666551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.666511    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6xlr\" (UniqueName: \"kubernetes.io/projected/826c9186-bbea-479d-a62c-968049a9651f-kube-api-access-d6xlr\") pod \"826c9186-bbea-479d-a62c-968049a9651f\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") "
Sep 17 19:34:10.666551 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.666543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826c9186-bbea-479d-a62c-968049a9651f-proxy-tls\") pod \"826c9186-bbea-479d-a62c-968049a9651f\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") "
Sep 17 19:34:10.666869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.666568    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-cabundle-cert\") pod \"826c9186-bbea-479d-a62c-968049a9651f\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") "
Sep 17 19:34:10.666869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.666593    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-kube-rbac-proxy-sar-config\") pod \"826c9186-bbea-479d-a62c-968049a9651f\" (UID: \"826c9186-bbea-479d-a62c-968049a9651f\") "
Sep 17 19:34:10.666869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.666824    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826c9186-bbea-479d-a62c-968049a9651f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "826c9186-bbea-479d-a62c-968049a9651f" (UID: "826c9186-bbea-479d-a62c-968049a9651f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:10.667038 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.666971    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "826c9186-bbea-479d-a62c-968049a9651f" (UID: "826c9186-bbea-479d-a62c-968049a9651f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:10.667038 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.666995    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "826c9186-bbea-479d-a62c-968049a9651f" (UID: "826c9186-bbea-479d-a62c-968049a9651f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:10.668849 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.668827    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826c9186-bbea-479d-a62c-968049a9651f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "826c9186-bbea-479d-a62c-968049a9651f" (UID: "826c9186-bbea-479d-a62c-968049a9651f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:10.668959 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.668846    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826c9186-bbea-479d-a62c-968049a9651f-kube-api-access-d6xlr" (OuterVolumeSpecName: "kube-api-access-d6xlr") pod "826c9186-bbea-479d-a62c-968049a9651f" (UID: "826c9186-bbea-479d-a62c-968049a9651f"). InnerVolumeSpecName "kube-api-access-d6xlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:10.767483 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.767440    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6xlr\" (UniqueName: \"kubernetes.io/projected/826c9186-bbea-479d-a62c-968049a9651f-kube-api-access-d6xlr\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:10.767483 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.767476    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/826c9186-bbea-479d-a62c-968049a9651f-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:10.767483 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.767489    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:10.767756 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.767500    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/826c9186-bbea-479d-a62c-968049a9651f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:10.767756 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.767511    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/826c9186-bbea-479d-a62c-968049a9651f-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:10.833261 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.833227    2583 generic.go:356] "Generic (PLEG): container finished" podID="826c9186-bbea-479d-a62c-968049a9651f" containerID="dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3" exitCode=0
Sep 17 19:34:10.833414 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.833286    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"
Sep 17 19:34:10.833414 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.833313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" event={"ID":"826c9186-bbea-479d-a62c-968049a9651f","Type":"ContainerDied","Data":"dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3"}
Sep 17 19:34:10.833490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.833427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" event={"ID":"826c9186-bbea-479d-a62c-968049a9651f","Type":"ContainerDied","Data":"be1ce5b782cdba5ac34bb2db8f6ff84f429fc7d8729acfd448dd095a19723103"}
Sep 17 19:34:10.833490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.833465    2583 scope.go:117] "RemoveContainer" containerID="f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0"
Sep 17 19:34:10.845663 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.845642    2583 scope.go:117] "RemoveContainer" containerID="dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3"
Sep 17 19:34:10.854302 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.854279    2583 scope.go:117] "RemoveContainer" containerID="39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d"
Sep 17 19:34:10.858970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.858946    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"]
Sep 17 19:34:10.861985 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.861960    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8"]
Sep 17 19:34:10.863066 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.863047    2583 scope.go:117] "RemoveContainer" containerID="f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0"
Sep 17 19:34:10.863342 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:34:10.863323    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0\": container with ID starting with f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0 not found: ID does not exist" containerID="f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0"
Sep 17 19:34:10.863396 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.863353    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0"} err="failed to get container status \"f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0\": rpc error: code = NotFound desc = could not find container \"f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0\": container with ID starting with f3721b3ead73f4490a4a582d56cf14259358b2b5074badf00dcdb3cffab3c2d0 not found: ID does not exist"
Sep 17 19:34:10.863396 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.863369    2583 scope.go:117] "RemoveContainer" containerID="dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3"
Sep 17 19:34:10.863602 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:34:10.863582    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3\": container with ID starting with dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3 not found: ID does not exist" containerID="dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3"
Sep 17 19:34:10.863660 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.863627    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3"} err="failed to get container status \"dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3\": rpc error: code = NotFound desc = could not find container \"dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3\": container with ID starting with dd00e0fb81181f3508028e34ba814f1d63326f7ae0a6f12f0d3b5d7f6717e3c3 not found: ID does not exist"
Sep 17 19:34:10.863660 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.863645    2583 scope.go:117] "RemoveContainer" containerID="39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d"
Sep 17 19:34:10.863847 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:34:10.863829    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d\": container with ID starting with 39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d not found: ID does not exist" containerID="39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d"
Sep 17 19:34:10.863887 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:10.863851    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d"} err="failed to get container status \"39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d\": rpc error: code = NotFound desc = could not find container \"39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d\": container with ID starting with 39cb4fef5109990ecb1e553e19d3fa30b21646393854d18af20e7e84eab6e68d not found: ID does not exist"
Sep 17 19:34:11.039029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:11.038999    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="826c9186-bbea-479d-a62c-968049a9651f" path="/var/lib/kubelet/pods/826c9186-bbea-479d-a62c-968049a9651f/volumes"
Sep 17 19:34:11.561679 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:11.561605    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8frn8" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": context deadline exceeded"
Sep 17 19:34:14.034204 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:14.034166    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:34:20.034552 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:20.034516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:34:23.178538 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:23.178506    2583 scope.go:117] "RemoveContainer" containerID="7ed59716bc8c5fc946bf8853c0d11fd36301bae5c6da3db147eab5991c113dd9"
Sep 17 19:34:24.034804 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:24.034780    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:34:30.271847 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.271813    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"]
Sep 17 19:34:30.272436 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.272090    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" containerID="cri-o://2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd" gracePeriod=30
Sep 17 19:34:30.272436 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.272105    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kube-rbac-proxy" containerID="cri-o://a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b" gracePeriod=30
Sep 17 19:34:30.276910 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.276880    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"]
Sep 17 19:34:30.277198 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.277148    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container" containerID="cri-o://5cf33029e0971b197b55075be454d4970893d0d352bc5d70a3613ca6c55aefcb" gracePeriod=30
Sep 17 19:34:30.277313 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.277247    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kube-rbac-proxy" containerID="cri-o://9662b3c9652460a298297e8a6146c595fb194b6280d2d30d0c6d04ffa2ed8cc8" gracePeriod=30
Sep 17 19:34:30.929056 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.929023    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerID="9662b3c9652460a298297e8a6146c595fb194b6280d2d30d0c6d04ffa2ed8cc8" exitCode=2
Sep 17 19:34:30.929236 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.929096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" event={"ID":"e4177a21-86ae-4281-b6b3-eae6117395ba","Type":"ContainerDied","Data":"9662b3c9652460a298297e8a6146c595fb194b6280d2d30d0c6d04ffa2ed8cc8"}
Sep 17 19:34:30.931165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.931146    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerID="a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b" exitCode=2
Sep 17 19:34:30.931252 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:30.931219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" event={"ID":"ec35817e-67ad-48cc-8836-4df13a6cb298","Type":"ContainerDied","Data":"a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b"}
Sep 17 19:34:32.487635 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:32.487544    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 17 19:34:33.495405 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:33.495366    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 19:34:34.034206 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.034167    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:34:34.923597 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.923564    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:34:34.952299 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.952273    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerID="5cf33029e0971b197b55075be454d4970893d0d352bc5d70a3613ca6c55aefcb" exitCode=0
Sep 17 19:34:34.952436 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.952346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" event={"ID":"e4177a21-86ae-4281-b6b3-eae6117395ba","Type":"ContainerDied","Data":"5cf33029e0971b197b55075be454d4970893d0d352bc5d70a3613ca6c55aefcb"}
Sep 17 19:34:34.954329 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.954307    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerID="2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd" exitCode=0
Sep 17 19:34:34.954458 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.954390    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"
Sep 17 19:34:34.954530 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.954389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" event={"ID":"ec35817e-67ad-48cc-8836-4df13a6cb298","Type":"ContainerDied","Data":"2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd"}
Sep 17 19:34:34.954530 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.954519    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7" event={"ID":"ec35817e-67ad-48cc-8836-4df13a6cb298","Type":"ContainerDied","Data":"9fac67c6cf61a1cceb12ea2145b5ea6f73dae95444f5a43aa81310293eff80a8"}
Sep 17 19:34:34.954656 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.954550    2583 scope.go:117] "RemoveContainer" containerID="a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b"
Sep 17 19:34:34.964555 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.964504    2583 scope.go:117] "RemoveContainer" containerID="2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd"
Sep 17 19:34:34.966702 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.966683    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec35817e-67ad-48cc-8836-4df13a6cb298-proxy-tls\") pod \"ec35817e-67ad-48cc-8836-4df13a6cb298\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") "
Sep 17 19:34:34.966806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.966714    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec35817e-67ad-48cc-8836-4df13a6cb298-kserve-provision-location\") pod \"ec35817e-67ad-48cc-8836-4df13a6cb298\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") "
Sep 17 19:34:34.966925 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.966905    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw46r\" (UniqueName: \"kubernetes.io/projected/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-api-access-bw46r\") pod \"ec35817e-67ad-48cc-8836-4df13a6cb298\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") "
Sep 17 19:34:34.966990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.966967    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-cabundle-cert\") pod \"ec35817e-67ad-48cc-8836-4df13a6cb298\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") "
Sep 17 19:34:34.967050 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.967022    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-rbac-proxy-sar-config\") pod \"ec35817e-67ad-48cc-8836-4df13a6cb298\" (UID: \"ec35817e-67ad-48cc-8836-4df13a6cb298\") "
Sep 17 19:34:34.967142 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.967019    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec35817e-67ad-48cc-8836-4df13a6cb298-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec35817e-67ad-48cc-8836-4df13a6cb298" (UID: "ec35817e-67ad-48cc-8836-4df13a6cb298"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:34.967315 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.967297    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec35817e-67ad-48cc-8836-4df13a6cb298-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:34.967406 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.967347    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec35817e-67ad-48cc-8836-4df13a6cb298" (UID: "ec35817e-67ad-48cc-8836-4df13a6cb298"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:34.967406 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.967366    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec35817e-67ad-48cc-8836-4df13a6cb298" (UID: "ec35817e-67ad-48cc-8836-4df13a6cb298"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:34.969007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.968982    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec35817e-67ad-48cc-8836-4df13a6cb298-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec35817e-67ad-48cc-8836-4df13a6cb298" (UID: "ec35817e-67ad-48cc-8836-4df13a6cb298"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:34.969540 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.969513    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-api-access-bw46r" (OuterVolumeSpecName: "kube-api-access-bw46r") pod "ec35817e-67ad-48cc-8836-4df13a6cb298" (UID: "ec35817e-67ad-48cc-8836-4df13a6cb298"). InnerVolumeSpecName "kube-api-access-bw46r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:34.986529 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.986504    2583 scope.go:117] "RemoveContainer" containerID="3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446"
Sep 17 19:34:34.995090 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.995070    2583 scope.go:117] "RemoveContainer" containerID="a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b"
Sep 17 19:34:34.995359 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:34:34.995340    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b\": container with ID starting with a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b not found: ID does not exist" containerID="a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b"
Sep 17 19:34:34.995415 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.995370    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b"} err="failed to get container status \"a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b\": rpc error: code = NotFound desc = could not find container \"a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b\": container with ID starting with a2f5d11d96e0a57c712b82c02ca5171b5208e839086f0018d98345c85348275b not found: ID does not exist"
Sep 17 19:34:34.995415 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.995389    2583 scope.go:117] "RemoveContainer" containerID="2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd"
Sep 17 19:34:34.995662 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:34:34.995646    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd\": container with ID starting with 2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd not found: ID does not exist" containerID="2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd"
Sep 17 19:34:34.995719 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.995669    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd"} err="failed to get container status \"2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd\": rpc error: code = NotFound desc = could not find container \"2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd\": container with ID starting with 2aeb9e26bb6e7c1363ae8dd92d71820cc3c5e2eb2819405f28873f5fd3f740cd not found: ID does not exist"
Sep 17 19:34:34.995719 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.995684    2583 scope.go:117] "RemoveContainer" containerID="3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446"
Sep 17 19:34:34.995958 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:34:34.995942    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446\": container with ID starting with 3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446 not found: ID does not exist" containerID="3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446"
Sep 17 19:34:34.996006 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:34.995965    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446"} err="failed to get container status \"3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446\": rpc error: code = NotFound desc = could not find container \"3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446\": container with ID starting with 3c8e04934e3cf661efe7458b0b87261663224c1145eabd2710892a63bff4e446 not found: ID does not exist"
Sep 17 19:34:35.015042 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.015025    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:34:35.068352 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068323    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4177a21-86ae-4281-b6b3-eae6117395ba-proxy-tls\") pod \"e4177a21-86ae-4281-b6b3-eae6117395ba\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") "
Sep 17 19:34:35.068534 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068382    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4177a21-86ae-4281-b6b3-eae6117395ba-kserve-provision-location\") pod \"e4177a21-86ae-4281-b6b3-eae6117395ba\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") "
Sep 17 19:34:35.068534 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068410    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-rbac-proxy-sar-config\") pod \"e4177a21-86ae-4281-b6b3-eae6117395ba\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") "
Sep 17 19:34:35.068534 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068470    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-cabundle-cert\") pod \"e4177a21-86ae-4281-b6b3-eae6117395ba\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") "
Sep 17 19:34:35.068534 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068512    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqcbf\" (UniqueName: \"kubernetes.io/projected/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-api-access-cqcbf\") pod \"e4177a21-86ae-4281-b6b3-eae6117395ba\" (UID: \"e4177a21-86ae-4281-b6b3-eae6117395ba\") "
Sep 17 19:34:35.068773 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068691    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4177a21-86ae-4281-b6b3-eae6117395ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4177a21-86ae-4281-b6b3-eae6117395ba" (UID: "e4177a21-86ae-4281-b6b3-eae6117395ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:35.068773 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068759    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4177a21-86ae-4281-b6b3-eae6117395ba" (UID: "e4177a21-86ae-4281-b6b3-eae6117395ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:35.068866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068837    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:35.068866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068838    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4177a21-86ae-4281-b6b3-eae6117395ba" (UID: "e4177a21-86ae-4281-b6b3-eae6117395ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:35.068866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068852    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:35.068866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068861    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec35817e-67ad-48cc-8836-4df13a6cb298-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:35.069001 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.068870    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw46r\" (UniqueName: \"kubernetes.io/projected/ec35817e-67ad-48cc-8836-4df13a6cb298-kube-api-access-bw46r\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:35.070482 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.070459    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4177a21-86ae-4281-b6b3-eae6117395ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4177a21-86ae-4281-b6b3-eae6117395ba" (UID: "e4177a21-86ae-4281-b6b3-eae6117395ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:35.070706 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.070685    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-api-access-cqcbf" (OuterVolumeSpecName: "kube-api-access-cqcbf") pod "e4177a21-86ae-4281-b6b3-eae6117395ba" (UID: "e4177a21-86ae-4281-b6b3-eae6117395ba"). InnerVolumeSpecName "kube-api-access-cqcbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:35.169587 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.169556    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4177a21-86ae-4281-b6b3-eae6117395ba-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:35.169587 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.169585    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4177a21-86ae-4281-b6b3-eae6117395ba-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:35.169587 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.169595    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:35.169767 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.169603    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4177a21-86ae-4281-b6b3-eae6117395ba-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:35.169767 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.169611    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqcbf\" (UniqueName: \"kubernetes.io/projected/e4177a21-86ae-4281-b6b3-eae6117395ba-kube-api-access-cqcbf\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:34:35.271370 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.271314    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"]
Sep 17 19:34:35.275272 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.275250    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-zjdl7"]
Sep 17 19:34:35.474069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474037    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"]
Sep 17 19:34:35.474374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474362    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container"
Sep 17 19:34:35.474419 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474377    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container"
Sep 17 19:34:35.474419 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474385    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kube-rbac-proxy"
Sep 17 19:34:35.474419 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474390    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kube-rbac-proxy"
Sep 17 19:34:35.474419 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474403    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="storage-initializer"
Sep 17 19:34:35.474419 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474408    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="storage-initializer"
Sep 17 19:34:35.474419 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474416    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="storage-initializer"
Sep 17 19:34:35.474419 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474420    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="storage-initializer"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474437    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kube-rbac-proxy"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474444    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kube-rbac-proxy"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474451    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="storage-initializer"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474456    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="storage-initializer"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474463    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kube-rbac-proxy"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474467    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kube-rbac-proxy"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474475    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kserve-container"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474480    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kserve-container"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474487    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474492    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474545    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kserve-container"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474555    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kserve-container"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474561    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" containerName="kube-rbac-proxy"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474567    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kube-rbac-proxy"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474573    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" containerName="kserve-container"
Sep 17 19:34:35.474645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.474578    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="826c9186-bbea-479d-a62c-968049a9651f" containerName="kube-rbac-proxy"
Sep 17 19:34:35.478083 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.478066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.480136 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.480119    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 19:34:35.480214 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.480157    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 19:34:35.487273 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.487250    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"]
Sep 17 19:34:35.514448 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.514426    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"]
Sep 17 19:34:35.518372 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.518357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.520720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.520699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 19:34:35.530002 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.529984    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"]
Sep 17 19:34:35.573219 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.573395 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.573395 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.573395 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4bf\" (UniqueName: \"kubernetes.io/projected/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-api-access-zf4bf\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.573395 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.573395 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573385    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86npk\" (UniqueName: \"kubernetes.io/projected/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-api-access-86npk\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.573686 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.573686 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573452    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.573686 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db1e2f5-96cc-4ca4-9508-e524684d08f7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.573686 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.573525    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.674894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.674856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.675069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.674895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.675069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.674931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.675069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.674967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4bf\" (UniqueName: \"kubernetes.io/projected/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-api-access-zf4bf\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.675069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.674993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.675069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86npk\" (UniqueName: \"kubernetes.io/projected/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-api-access-86npk\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.675069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.675443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.675443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.675443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db1e2f5-96cc-4ca4-9508-e524684d08f7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.675443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.675735 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675711    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.675871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.675871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.675871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.675841    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.676063 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.676046    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.677812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.677795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.677878 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.677864    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db1e2f5-96cc-4ca4-9508-e524684d08f7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.682869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.682851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4bf\" (UniqueName: \"kubernetes.io/projected/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-api-access-zf4bf\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.682959 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.682930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86npk\" (UniqueName: \"kubernetes.io/projected/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-api-access-86npk\") pod \"isvc-canary-promote-predictor-78b4c78fb9-qjxj7\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.788789 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.788733    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:35.827418 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.827278    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:35.961100 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.961074    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"
Sep 17 19:34:35.961552 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.961069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8" event={"ID":"e4177a21-86ae-4281-b6b3-eae6117395ba","Type":"ContainerDied","Data":"80385ccedfb15a1bc54d671825993e7b5536f529110fd4978955f5b26f3c7e90"}
Sep 17 19:34:35.961552 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.961232    2583 scope.go:117] "RemoveContainer" containerID="9662b3c9652460a298297e8a6146c595fb194b6280d2d30d0c6d04ffa2ed8cc8"
Sep 17 19:34:35.969772 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.969754    2583 scope.go:117] "RemoveContainer" containerID="5cf33029e0971b197b55075be454d4970893d0d352bc5d70a3613ca6c55aefcb"
Sep 17 19:34:35.977608 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.977593    2583 scope.go:117] "RemoveContainer" containerID="de8a33a1b3e45543f9311bbc2befb90cc8a597f0a5d6327d2cff7085bd24794e"
Sep 17 19:34:35.982189 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.982166    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"]
Sep 17 19:34:35.988007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:35.987986    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-tjpp8"]
Sep 17 19:34:36.067575 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:36.067552    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"]
Sep 17 19:34:36.070127 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:34:36.070100    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db1e2f5_96cc_4ca4_9508_e524684d08f7.slice/crio-1123eba2caab775a9073dbdb14e86426cf8cd6f7e76ea2776e838293a0c247e9 WatchSource:0}: Error finding container 1123eba2caab775a9073dbdb14e86426cf8cd6f7e76ea2776e838293a0c247e9: Status 404 returned error can't find the container with id 1123eba2caab775a9073dbdb14e86426cf8cd6f7e76ea2776e838293a0c247e9
Sep 17 19:34:36.111785 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:36.111762    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"]
Sep 17 19:34:36.114801 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:34:36.114772    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode224c4bb_22fa_4d4e_bd43_e95fc13083ac.slice/crio-becd7e993d96da0f5c0fd8a26ccb3a16c67dd1a8b19c47b3ffba377f54228c44 WatchSource:0}: Error finding container becd7e993d96da0f5c0fd8a26ccb3a16c67dd1a8b19c47b3ffba377f54228c44: Status 404 returned error can't find the container with id becd7e993d96da0f5c0fd8a26ccb3a16c67dd1a8b19c47b3ffba377f54228c44
Sep 17 19:34:36.968510 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:36.968473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" event={"ID":"8db1e2f5-96cc-4ca4-9508-e524684d08f7","Type":"ContainerStarted","Data":"eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6"}
Sep 17 19:34:36.969009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:36.968537    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" event={"ID":"8db1e2f5-96cc-4ca4-9508-e524684d08f7","Type":"ContainerStarted","Data":"1123eba2caab775a9073dbdb14e86426cf8cd6f7e76ea2776e838293a0c247e9"}
Sep 17 19:34:36.969954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:36.969933    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" event={"ID":"e224c4bb-22fa-4d4e-bd43-e95fc13083ac","Type":"ContainerStarted","Data":"69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70"}
Sep 17 19:34:36.970047 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:36.969979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" event={"ID":"e224c4bb-22fa-4d4e-bd43-e95fc13083ac","Type":"ContainerStarted","Data":"becd7e993d96da0f5c0fd8a26ccb3a16c67dd1a8b19c47b3ffba377f54228c44"}
Sep 17 19:34:37.039240 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:37.039211    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4177a21-86ae-4281-b6b3-eae6117395ba" path="/var/lib/kubelet/pods/e4177a21-86ae-4281-b6b3-eae6117395ba/volumes"
Sep 17 19:34:37.039795 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:37.039780    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec35817e-67ad-48cc-8836-4df13a6cb298" path="/var/lib/kubelet/pods/ec35817e-67ad-48cc-8836-4df13a6cb298/volumes"
Sep 17 19:34:39.987395 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:39.987360    2583 generic.go:356] "Generic (PLEG): container finished" podID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerID="eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6" exitCode=0
Sep 17 19:34:39.987738 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:39.987438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" event={"ID":"8db1e2f5-96cc-4ca4-9508-e524684d08f7","Type":"ContainerDied","Data":"eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6"}
Sep 17 19:34:40.993700 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:40.993670    2583 generic.go:356] "Generic (PLEG): container finished" podID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerID="69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70" exitCode=0
Sep 17 19:34:40.994109 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:40.993750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" event={"ID":"e224c4bb-22fa-4d4e-bd43-e95fc13083ac","Type":"ContainerDied","Data":"69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70"}
Sep 17 19:34:40.995879 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:40.995856    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" event={"ID":"8db1e2f5-96cc-4ca4-9508-e524684d08f7","Type":"ContainerStarted","Data":"ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc"}
Sep 17 19:34:40.995989 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:40.995902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" event={"ID":"8db1e2f5-96cc-4ca4-9508-e524684d08f7","Type":"ContainerStarted","Data":"9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5"}
Sep 17 19:34:40.996164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:40.996137    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:40.996227 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:40.996187    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:40.997320 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:40.997290    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:34:41.037029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:41.036988    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podStartSLOduration=2.239059589 podStartE2EDuration="6.036967909s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.79790832s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:41.035442236 +0000 UTC m=+1700.163183650" watchObservedRunningTime="2026-09-17 19:34:41.036967909 +0000 UTC m=+1700.164709311"
Sep 17 19:34:42.003455 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:42.003423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" event={"ID":"e224c4bb-22fa-4d4e-bd43-e95fc13083ac","Type":"ContainerStarted","Data":"1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57"}
Sep 17 19:34:42.003984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:42.003483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" event={"ID":"e224c4bb-22fa-4d4e-bd43-e95fc13083ac","Type":"ContainerStarted","Data":"676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab"}
Sep 17 19:34:42.003984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:42.003745    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:34:42.003984 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:42.003864    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:42.004127 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:42.004028    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:42.005266 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:42.005241    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:34:42.023209 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:42.023162    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podStartSLOduration=2.832955228 podStartE2EDuration="7.023144465s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.190189237s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:42.02205192 +0000 UTC m=+1701.149793332" watchObservedRunningTime="2026-09-17 19:34:42.023144465 +0000 UTC m=+1701.150885866"
Sep 17 19:34:43.008626 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:43.008565    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:34:47.009415 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:47.009389    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:34:47.009938 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:47.009907    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:34:48.012691 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:48.012660    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:34:48.013324 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:48.013298    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:34:57.009975 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:57.009940    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:34:58.013715 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:34:58.013672    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:35:07.009937 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:07.009895    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:35:08.013398 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:08.013359    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:35:17.009811 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:17.009769    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:35:18.013327 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:18.013291    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:35:23.196166 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:23.196134    2583 scope.go:117] "RemoveContainer" containerID="3d967a7517441e4bd4f3654ad644311e19826b6636f4a50a50a1075a1fea358d"
Sep 17 19:35:23.204483 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:23.204462    2583 scope.go:117] "RemoveContainer" containerID="a81af3335b994d101d734a9275ac8e9ea744efc8ffb7973ba6b519a8999ffd28"
Sep 17 19:35:27.010703 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:27.010666    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:35:28.013209 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:28.013167    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:35:37.010835 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:37.010789    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:35:38.013914 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:38.013875    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:35:44.145744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.145708    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"]
Sep 17 19:35:44.148958 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.148941    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.152361 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.152334    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 19:35:44.152479 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.152358    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 19:35:44.177987 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.177967    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"]
Sep 17 19:35:44.255165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.255129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.255303 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.255177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vltx\" (UniqueName: \"kubernetes.io/projected/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-api-access-4vltx\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.255303 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.255252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.255303 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.255282    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.255427 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.255323    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.356577 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.356545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.356577 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.356583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vltx\" (UniqueName: \"kubernetes.io/projected/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-api-access-4vltx\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.356778 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.356608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.356778 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.356650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.356778 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.356692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.356918 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:35:44.356817    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 17 19:35:44.356918 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:35:44.356897    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-proxy-tls podName:7cfccd16-2f26-4ba4-b268-32faf01fe7ea nodeName:}" failed. No retries permitted until 2026-09-17 19:35:44.856876029 +0000 UTC m=+1763.984617411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" (UID: "7cfccd16-2f26-4ba4-b268-32faf01fe7ea") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 17 19:35:44.357027 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.357007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.357293 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.357278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.357349 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.357333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.365657 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.365637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vltx\" (UniqueName: \"kubernetes.io/projected/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-api-access-4vltx\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.860061 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.860023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:44.862528 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:44.862502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:45.059195 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:45.059161    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:45.333294 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:35:45.333249    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfccd16_2f26_4ba4_b268_32faf01fe7ea.slice/crio-b86f656cf20e89efce5399eb27eb9fbdc26e702fdbb8e57a4b1b0ec9299359fd WatchSource:0}: Error finding container b86f656cf20e89efce5399eb27eb9fbdc26e702fdbb8e57a4b1b0ec9299359fd: Status 404 returned error can't find the container with id b86f656cf20e89efce5399eb27eb9fbdc26e702fdbb8e57a4b1b0ec9299359fd
Sep 17 19:35:45.335294 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:45.335271    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"]
Sep 17 19:35:46.286730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:46.286696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" event={"ID":"7cfccd16-2f26-4ba4-b268-32faf01fe7ea","Type":"ContainerStarted","Data":"23a530038ee7f401b56bdfde60221567ecf64b1dec4e6ce42f6fac46dd69edad"}
Sep 17 19:35:46.286933 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:46.286796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" event={"ID":"7cfccd16-2f26-4ba4-b268-32faf01fe7ea","Type":"ContainerStarted","Data":"b86f656cf20e89efce5399eb27eb9fbdc26e702fdbb8e57a4b1b0ec9299359fd"}
Sep 17 19:35:47.009995 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:47.009959    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:35:48.013307 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:48.013270    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:35:50.304883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:50.304854    2583 generic.go:356] "Generic (PLEG): container finished" podID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerID="23a530038ee7f401b56bdfde60221567ecf64b1dec4e6ce42f6fac46dd69edad" exitCode=0
Sep 17 19:35:50.305260 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:50.304935    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" event={"ID":"7cfccd16-2f26-4ba4-b268-32faf01fe7ea","Type":"ContainerDied","Data":"23a530038ee7f401b56bdfde60221567ecf64b1dec4e6ce42f6fac46dd69edad"}
Sep 17 19:35:51.310679 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:51.310644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" event={"ID":"7cfccd16-2f26-4ba4-b268-32faf01fe7ea","Type":"ContainerStarted","Data":"b86d3354a6549b81810a34931b8e9d3b80808595e9054a0ccafd4e8dc49707d9"}
Sep 17 19:35:51.311083 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:51.310709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" event={"ID":"7cfccd16-2f26-4ba4-b268-32faf01fe7ea","Type":"ContainerStarted","Data":"bdd1efc9633da283cfca61ca4c6cadefa2409d19d2fc2dfd0059c345ada265ff"}
Sep 17 19:35:51.311083 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:51.310961    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:51.311170 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:51.311102    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:51.312290 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:51.312265    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:35:51.329596 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:51.329556    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podStartSLOduration=3.34891007 podStartE2EDuration="7.329544355s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.980634285s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:35:51.328467484 +0000 UTC m=+1770.456208884" watchObservedRunningTime="2026-09-17 19:35:51.329544355 +0000 UTC m=+1770.457285756"
Sep 17 19:35:52.315567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:52.315526    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:35:57.010743 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:57.010710    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:35:57.134897 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:57.134860    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"]
Sep 17 19:35:57.135146 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:57.135095    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container" containerID="cri-o://676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab" gracePeriod=30
Sep 17 19:35:57.135298 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:57.135169    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kube-rbac-proxy" containerID="cri-o://1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57" gracePeriod=30
Sep 17 19:35:57.135953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:57.135777    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:35:57.320099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:57.320037    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:35:57.320552 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:57.320528    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:35:57.336578 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:57.336548    2583 generic.go:356] "Generic (PLEG): container finished" podID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerID="1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57" exitCode=2
Sep 17 19:35:57.336716 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:57.336603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" event={"ID":"e224c4bb-22fa-4d4e-bd43-e95fc13083ac","Type":"ContainerDied","Data":"1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57"}
Sep 17 19:35:58.009482 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:35:58.009437    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 17 19:36:02.175374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.175344    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:36:02.309965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.309885    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kserve-provision-location\") pod \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") "
Sep 17 19:36:02.309965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.309929    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-rbac-proxy-sar-config\") pod \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") "
Sep 17 19:36:02.309965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.309962    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-cabundle-cert\") pod \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") "
Sep 17 19:36:02.310225 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.309993    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86npk\" (UniqueName: \"kubernetes.io/projected/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-api-access-86npk\") pod \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") "
Sep 17 19:36:02.310225 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.310039    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-proxy-tls\") pod \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\" (UID: \"e224c4bb-22fa-4d4e-bd43-e95fc13083ac\") "
Sep 17 19:36:02.310355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.310294    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e224c4bb-22fa-4d4e-bd43-e95fc13083ac" (UID: "e224c4bb-22fa-4d4e-bd43-e95fc13083ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:36:02.310355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.310303    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e224c4bb-22fa-4d4e-bd43-e95fc13083ac" (UID: "e224c4bb-22fa-4d4e-bd43-e95fc13083ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:36:02.310458 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.310371    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e224c4bb-22fa-4d4e-bd43-e95fc13083ac" (UID: "e224c4bb-22fa-4d4e-bd43-e95fc13083ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:36:02.312306 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.312287    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e224c4bb-22fa-4d4e-bd43-e95fc13083ac" (UID: "e224c4bb-22fa-4d4e-bd43-e95fc13083ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:36:02.312407 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.312385    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-api-access-86npk" (OuterVolumeSpecName: "kube-api-access-86npk") pod "e224c4bb-22fa-4d4e-bd43-e95fc13083ac" (UID: "e224c4bb-22fa-4d4e-bd43-e95fc13083ac"). InnerVolumeSpecName "kube-api-access-86npk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:36:02.360076 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.360041    2583 generic.go:356] "Generic (PLEG): container finished" podID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerID="676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab" exitCode=0
Sep 17 19:36:02.360257 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.360093    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" event={"ID":"e224c4bb-22fa-4d4e-bd43-e95fc13083ac","Type":"ContainerDied","Data":"676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab"}
Sep 17 19:36:02.360257 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.360100    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"
Sep 17 19:36:02.360257 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.360152    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7" event={"ID":"e224c4bb-22fa-4d4e-bd43-e95fc13083ac","Type":"ContainerDied","Data":"becd7e993d96da0f5c0fd8a26ccb3a16c67dd1a8b19c47b3ffba377f54228c44"}
Sep 17 19:36:02.360257 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.360181    2583 scope.go:117] "RemoveContainer" containerID="1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57"
Sep 17 19:36:02.369045 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.369030    2583 scope.go:117] "RemoveContainer" containerID="676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab"
Sep 17 19:36:02.376784 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.376769    2583 scope.go:117] "RemoveContainer" containerID="69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70"
Sep 17 19:36:02.380931 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.380908    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"]
Sep 17 19:36:02.384146 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.384125    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-qjxj7"]
Sep 17 19:36:02.384809 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.384793    2583 scope.go:117] "RemoveContainer" containerID="1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57"
Sep 17 19:36:02.385093 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:36:02.385077    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57\": container with ID starting with 1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57 not found: ID does not exist" containerID="1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57"
Sep 17 19:36:02.385148 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.385100    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57"} err="failed to get container status \"1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57\": rpc error: code = NotFound desc = could not find container \"1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57\": container with ID starting with 1c4a7ab3b45442df76a4d320e1065e5f3c14f3f30b92ef4eb4ec8382d4e7fe57 not found: ID does not exist"
Sep 17 19:36:02.385148 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.385117    2583 scope.go:117] "RemoveContainer" containerID="676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab"
Sep 17 19:36:02.385384 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:36:02.385366    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab\": container with ID starting with 676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab not found: ID does not exist" containerID="676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab"
Sep 17 19:36:02.385454 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.385393    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab"} err="failed to get container status \"676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab\": rpc error: code = NotFound desc = could not find container \"676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab\": container with ID starting with 676a0a6cac3c75134c88222c86a91e290b4c5b92f07f2a6ffe1a44d24a4535ab not found: ID does not exist"
Sep 17 19:36:02.385454 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.385414    2583 scope.go:117] "RemoveContainer" containerID="69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70"
Sep 17 19:36:02.385663 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:36:02.385646    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70\": container with ID starting with 69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70 not found: ID does not exist" containerID="69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70"
Sep 17 19:36:02.385713 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.385670    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70"} err="failed to get container status \"69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70\": rpc error: code = NotFound desc = could not find container \"69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70\": container with ID starting with 69dbb5f72b3797c49687e5fdae1b6bc6899817dde38f9d9ddd4821ff62632c70 not found: ID does not exist"
Sep 17 19:36:02.411116 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.411098    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:36:02.411116 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.411117    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:36:02.411229 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.411126    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:36:02.411229 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.411136    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:36:02.411229 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:02.411143    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86npk\" (UniqueName: \"kubernetes.io/projected/e224c4bb-22fa-4d4e-bd43-e95fc13083ac-kube-api-access-86npk\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:36:03.038655 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:03.038607    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" path="/var/lib/kubelet/pods/e224c4bb-22fa-4d4e-bd43-e95fc13083ac/volumes"
Sep 17 19:36:07.320697 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:07.320656    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:36:17.320939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:17.320898    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:36:27.320947 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:27.320903    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:36:27.784895 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:27.784861    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:36:27.786105 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:27.786082    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:36:37.321443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:37.321400    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:36:47.321030 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:47.320992    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:36:57.320757 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:36:57.320716    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:37:07.038440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:07.038403    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:37:14.880426 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:14.880394    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"]
Sep 17 19:37:14.880848 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:14.880630    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" containerID="cri-o://bdd1efc9633da283cfca61ca4c6cadefa2409d19d2fc2dfd0059c345ada265ff" gracePeriod=30
Sep 17 19:37:14.880848 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:14.880686    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kube-rbac-proxy" containerID="cri-o://b86d3354a6549b81810a34931b8e9d3b80808595e9054a0ccafd4e8dc49707d9" gracePeriod=30
Sep 17 19:37:15.679117 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:15.679080    2583 generic.go:356] "Generic (PLEG): container finished" podID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerID="b86d3354a6549b81810a34931b8e9d3b80808595e9054a0ccafd4e8dc49707d9" exitCode=2
Sep 17 19:37:15.679294 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:15.679152    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" event={"ID":"7cfccd16-2f26-4ba4-b268-32faf01fe7ea","Type":"ContainerDied","Data":"b86d3354a6549b81810a34931b8e9d3b80808595e9054a0ccafd4e8dc49707d9"}
Sep 17 19:37:17.033914 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:17.033875    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:37:17.316065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:17.315972    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 19:37:19.698928 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.698868    2583 generic.go:356] "Generic (PLEG): container finished" podID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerID="bdd1efc9633da283cfca61ca4c6cadefa2409d19d2fc2dfd0059c345ada265ff" exitCode=0
Sep 17 19:37:19.699175 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.698938    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" event={"ID":"7cfccd16-2f26-4ba4-b268-32faf01fe7ea","Type":"ContainerDied","Data":"bdd1efc9633da283cfca61ca4c6cadefa2409d19d2fc2dfd0059c345ada265ff"}
Sep 17 19:37:19.824869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.824846    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:37:19.947793 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.947763    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-rbac-proxy-sar-config\") pod \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") "
Sep 17 19:37:19.947970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.947801    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-cabundle-cert\") pod \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") "
Sep 17 19:37:19.947970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.947846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-proxy-tls\") pod \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") "
Sep 17 19:37:19.947970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.947952    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kserve-provision-location\") pod \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") "
Sep 17 19:37:19.948140 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.948025    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vltx\" (UniqueName: \"kubernetes.io/projected/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-api-access-4vltx\") pod \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\" (UID: \"7cfccd16-2f26-4ba4-b268-32faf01fe7ea\") "
Sep 17 19:37:19.948314 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.948286    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7cfccd16-2f26-4ba4-b268-32faf01fe7ea" (UID: "7cfccd16-2f26-4ba4-b268-32faf01fe7ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:37:19.948314 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.948299    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7cfccd16-2f26-4ba4-b268-32faf01fe7ea" (UID: "7cfccd16-2f26-4ba4-b268-32faf01fe7ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:19.948448 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.948292    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7cfccd16-2f26-4ba4-b268-32faf01fe7ea" (UID: "7cfccd16-2f26-4ba4-b268-32faf01fe7ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:37:19.950091 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.950066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cfccd16-2f26-4ba4-b268-32faf01fe7ea" (UID: "7cfccd16-2f26-4ba4-b268-32faf01fe7ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:37:19.950197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:19.950174    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-api-access-4vltx" (OuterVolumeSpecName: "kube-api-access-4vltx") pod "7cfccd16-2f26-4ba4-b268-32faf01fe7ea" (UID: "7cfccd16-2f26-4ba4-b268-32faf01fe7ea"). InnerVolumeSpecName "kube-api-access-4vltx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:37:20.049562 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.049520    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:20.049562 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.049554    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:20.049562 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.049564    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:20.049562 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.049572    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:20.049860 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.049581    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vltx\" (UniqueName: \"kubernetes.io/projected/7cfccd16-2f26-4ba4-b268-32faf01fe7ea-kube-api-access-4vltx\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:20.071755 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.071723    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"]
Sep 17 19:37:20.072114 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072097    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container"
Sep 17 19:37:20.072197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072116    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container"
Sep 17 19:37:20.072197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072143    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kube-rbac-proxy"
Sep 17 19:37:20.072197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072151    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kube-rbac-proxy"
Sep 17 19:37:20.072197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072162    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kube-rbac-proxy"
Sep 17 19:37:20.072197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072170    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kube-rbac-proxy"
Sep 17 19:37:20.072197 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072194    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="storage-initializer"
Sep 17 19:37:20.072516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072202    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="storage-initializer"
Sep 17 19:37:20.072516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072216    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="storage-initializer"
Sep 17 19:37:20.072516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072225    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="storage-initializer"
Sep 17 19:37:20.072516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072238    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container"
Sep 17 19:37:20.072516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072247    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container"
Sep 17 19:37:20.072516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072318    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kserve-container"
Sep 17 19:37:20.072516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072333    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e224c4bb-22fa-4d4e-bd43-e95fc13083ac" containerName="kube-rbac-proxy"
Sep 17 19:37:20.072516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072343    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kube-rbac-proxy"
Sep 17 19:37:20.072516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.072353    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" containerName="kserve-container"
Sep 17 19:37:20.075902 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.075880    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.078370 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.078351    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 19:37:20.078486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.078353    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 19:37:20.087501 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.087476    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"]
Sep 17 19:37:20.150918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.150882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fltbc\" (UniqueName: \"kubernetes.io/projected/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-api-access-fltbc\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.150918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.150916    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.151181 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.150937    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a5ee734-03f4-46a8-9210-f718142ba5ba-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.151181 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.151069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.151181 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.151112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5ee734-03f4-46a8-9210-f718142ba5ba-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.252314 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.252238    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.252314 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.252271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5ee734-03f4-46a8-9210-f718142ba5ba-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.252314 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.252308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fltbc\" (UniqueName: \"kubernetes.io/projected/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-api-access-fltbc\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.252586 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.252327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.252586 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.252348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a5ee734-03f4-46a8-9210-f718142ba5ba-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.252779 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.252756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5ee734-03f4-46a8-9210-f718142ba5ba-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.253155 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.253132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.253201 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.253132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.254850 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.254832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a5ee734-03f4-46a8-9210-f718142ba5ba-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.261225 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.261199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fltbc\" (UniqueName: \"kubernetes.io/projected/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-api-access-fltbc\") pod \"isvc-tensorflow-predictor-547dfc5669-wtmtq\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.385820 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.385786    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:20.658543 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.658460    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"]
Sep 17 19:37:20.661891 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:37:20.661864    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5ee734_03f4_46a8_9210_f718142ba5ba.slice/crio-30ed1a5dfb960cca95e563dcbcf3e8383a58dc12614703e073fd86be8e967a90 WatchSource:0}: Error finding container 30ed1a5dfb960cca95e563dcbcf3e8383a58dc12614703e073fd86be8e967a90: Status 404 returned error can't find the container with id 30ed1a5dfb960cca95e563dcbcf3e8383a58dc12614703e073fd86be8e967a90
Sep 17 19:37:20.705008 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.704987    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"
Sep 17 19:37:20.705349 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.705010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx" event={"ID":"7cfccd16-2f26-4ba4-b268-32faf01fe7ea","Type":"ContainerDied","Data":"b86f656cf20e89efce5399eb27eb9fbdc26e702fdbb8e57a4b1b0ec9299359fd"}
Sep 17 19:37:20.705349 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.705097    2583 scope.go:117] "RemoveContainer" containerID="b86d3354a6549b81810a34931b8e9d3b80808595e9054a0ccafd4e8dc49707d9"
Sep 17 19:37:20.706533 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.706501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" event={"ID":"8a5ee734-03f4-46a8-9210-f718142ba5ba","Type":"ContainerStarted","Data":"30ed1a5dfb960cca95e563dcbcf3e8383a58dc12614703e073fd86be8e967a90"}
Sep 17 19:37:20.713756 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.713628    2583 scope.go:117] "RemoveContainer" containerID="bdd1efc9633da283cfca61ca4c6cadefa2409d19d2fc2dfd0059c345ada265ff"
Sep 17 19:37:20.722662 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.722645    2583 scope.go:117] "RemoveContainer" containerID="23a530038ee7f401b56bdfde60221567ecf64b1dec4e6ce42f6fac46dd69edad"
Sep 17 19:37:20.731523 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.731506    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"]
Sep 17 19:37:20.736359 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:20.736339    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-ljrgx"]
Sep 17 19:37:21.038796 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:21.038753    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cfccd16-2f26-4ba4-b268-32faf01fe7ea" path="/var/lib/kubelet/pods/7cfccd16-2f26-4ba4-b268-32faf01fe7ea/volumes"
Sep 17 19:37:21.712632 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:21.712588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" event={"ID":"8a5ee734-03f4-46a8-9210-f718142ba5ba","Type":"ContainerStarted","Data":"72440f2f58e994bf6487da71f1a58b9caedd6d73baf393fc036107eedd02f90b"}
Sep 17 19:37:26.736589 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:26.736551    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerID="72440f2f58e994bf6487da71f1a58b9caedd6d73baf393fc036107eedd02f90b" exitCode=0
Sep 17 19:37:26.737047 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:26.736649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" event={"ID":"8a5ee734-03f4-46a8-9210-f718142ba5ba","Type":"ContainerDied","Data":"72440f2f58e994bf6487da71f1a58b9caedd6d73baf393fc036107eedd02f90b"}
Sep 17 19:37:26.738003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:26.737988    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:37:31.769019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:31.768975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" event={"ID":"8a5ee734-03f4-46a8-9210-f718142ba5ba","Type":"ContainerStarted","Data":"e27cbec1d1fb5c2dcdd1eb8e892a84f15f00956bdaa874351521dce10fb96268"}
Sep 17 19:37:31.769486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:31.769054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" event={"ID":"8a5ee734-03f4-46a8-9210-f718142ba5ba","Type":"ContainerStarted","Data":"57f7f0b0fe4dddafa8c5e239cd2e1ba88edc856ab262759050653b26c20114c1"}
Sep 17 19:37:31.769486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:31.769264    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:31.789799 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:31.789751    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podStartSLOduration=2.645457583 podStartE2EDuration="11.789736867s" totalImagesPullingTime="4.148917987s" totalInitContainerRuntime="4.995361297s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:37:31.78793195 +0000 UTC m=+1870.915673351" watchObservedRunningTime="2026-09-17 19:37:31.789736867 +0000 UTC m=+1870.917478267"
Sep 17 19:37:32.774180 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:32.774149    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:32.775314 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:32.775285    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:37:33.777852 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:33.777808    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:37:35.694763 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:35.694716    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"]
Sep 17 19:37:35.695231 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:35.694951    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" containerID="cri-o://9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5" gracePeriod=30
Sep 17 19:37:35.695231 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:35.695034    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kube-rbac-proxy" containerID="cri-o://ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc" gracePeriod=30
Sep 17 19:37:36.791787 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:36.791751    2583 generic.go:356] "Generic (PLEG): container finished" podID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerID="ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc" exitCode=2
Sep 17 19:37:36.792181 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:36.791822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" event={"ID":"8db1e2f5-96cc-4ca4-9508-e524684d08f7","Type":"ContainerDied","Data":"ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc"}
Sep 17 19:37:37.004325 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:37.004282    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 19:37:37.010763 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:37.010728    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:37:38.789223 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:38.789195    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:38.789758 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:38.789721    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:37:40.743334 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.743297    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:37:40.810873 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.810789    2583 generic.go:356] "Generic (PLEG): container finished" podID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerID="9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5" exitCode=0
Sep 17 19:37:40.810873 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.810861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" event={"ID":"8db1e2f5-96cc-4ca4-9508-e524684d08f7","Type":"ContainerDied","Data":"9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5"}
Sep 17 19:37:40.810873 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.810869    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"
Sep 17 19:37:40.811149 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.810950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl" event={"ID":"8db1e2f5-96cc-4ca4-9508-e524684d08f7","Type":"ContainerDied","Data":"1123eba2caab775a9073dbdb14e86426cf8cd6f7e76ea2776e838293a0c247e9"}
Sep 17 19:37:40.811149 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.810987    2583 scope.go:117] "RemoveContainer" containerID="ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc"
Sep 17 19:37:40.819766 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.819745    2583 scope.go:117] "RemoveContainer" containerID="9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5"
Sep 17 19:37:40.827480 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.827460    2583 scope.go:117] "RemoveContainer" containerID="eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6"
Sep 17 19:37:40.835009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.834994    2583 scope.go:117] "RemoveContainer" containerID="ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc"
Sep 17 19:37:40.835295 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:37:40.835273    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc\": container with ID starting with ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc not found: ID does not exist" containerID="ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc"
Sep 17 19:37:40.835343 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835305    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc"} err="failed to get container status \"ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc\": rpc error: code = NotFound desc = could not find container \"ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc\": container with ID starting with ceabf5619ca0fe4d0eeae5eede15e9337f77b7572d29a3df164b3f65a84685cc not found: ID does not exist"
Sep 17 19:37:40.835343 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835323    2583 scope.go:117] "RemoveContainer" containerID="9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5"
Sep 17 19:37:40.835558 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:37:40.835541    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5\": container with ID starting with 9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5 not found: ID does not exist" containerID="9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5"
Sep 17 19:37:40.835639 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835569    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5"} err="failed to get container status \"9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5\": rpc error: code = NotFound desc = could not find container \"9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5\": container with ID starting with 9d4c13a68bdf6baaf2821e597f077a11f52bb5833b13a29a835817a0aea198b5 not found: ID does not exist"
Sep 17 19:37:40.835639 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835595    2583 scope.go:117] "RemoveContainer" containerID="eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6"
Sep 17 19:37:40.835757 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kserve-provision-location\") pod \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") "
Sep 17 19:37:40.835812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835755    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf4bf\" (UniqueName: \"kubernetes.io/projected/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-api-access-zf4bf\") pod \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") "
Sep 17 19:37:40.835866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835835    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-cabundle-cert\") pod \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") "
Sep 17 19:37:40.835866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835844    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8db1e2f5-96cc-4ca4-9508-e524684d08f7" (UID: "8db1e2f5-96cc-4ca4-9508-e524684d08f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:40.835976 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835879    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db1e2f5-96cc-4ca4-9508-e524684d08f7-proxy-tls\") pod \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") "
Sep 17 19:37:40.835976 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835908    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-rbac-proxy-sar-config\") pod \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\" (UID: \"8db1e2f5-96cc-4ca4-9508-e524684d08f7\") "
Sep 17 19:37:40.835976 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:37:40.835930    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6\": container with ID starting with eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6 not found: ID does not exist" containerID="eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6"
Sep 17 19:37:40.835976 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.835964    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6"} err="failed to get container status \"eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6\": rpc error: code = NotFound desc = could not find container \"eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6\": container with ID starting with eb55d775ee7893a8652f176e1cf0dde43d2bbd8f7084abad03e6686b973bcdc6 not found: ID does not exist"
Sep 17 19:37:40.836169 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.836151    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:40.836212 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.836165    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8db1e2f5-96cc-4ca4-9508-e524684d08f7" (UID: "8db1e2f5-96cc-4ca4-9508-e524684d08f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:37:40.836321 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.836302    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8db1e2f5-96cc-4ca4-9508-e524684d08f7" (UID: "8db1e2f5-96cc-4ca4-9508-e524684d08f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:37:40.837872 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.837844    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-api-access-zf4bf" (OuterVolumeSpecName: "kube-api-access-zf4bf") pod "8db1e2f5-96cc-4ca4-9508-e524684d08f7" (UID: "8db1e2f5-96cc-4ca4-9508-e524684d08f7"). InnerVolumeSpecName "kube-api-access-zf4bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:37:40.838055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.838040    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db1e2f5-96cc-4ca4-9508-e524684d08f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8db1e2f5-96cc-4ca4-9508-e524684d08f7" (UID: "8db1e2f5-96cc-4ca4-9508-e524684d08f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:37:40.937212 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.937184    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db1e2f5-96cc-4ca4-9508-e524684d08f7-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:40.937212 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.937210    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:40.937404 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.937221    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf4bf\" (UniqueName: \"kubernetes.io/projected/8db1e2f5-96cc-4ca4-9508-e524684d08f7-kube-api-access-zf4bf\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:40.937404 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:40.937230    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db1e2f5-96cc-4ca4-9508-e524684d08f7-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:37:41.002171 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.002136    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"]
Sep 17 19:37:41.002465 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.002453    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kube-rbac-proxy"
Sep 17 19:37:41.002508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.002466    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kube-rbac-proxy"
Sep 17 19:37:41.002508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.002479    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container"
Sep 17 19:37:41.002508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.002485    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container"
Sep 17 19:37:41.002508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.002496    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="storage-initializer"
Sep 17 19:37:41.002508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.002503    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="storage-initializer"
Sep 17 19:37:41.002683 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.002552    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kube-rbac-proxy"
Sep 17 19:37:41.002683 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.002561    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" containerName="kserve-container"
Sep 17 19:37:41.005696 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.005680    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.008296 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.008268    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 19:37:41.008296 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.008287    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 19:37:41.022925 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.022900    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"]
Sep 17 19:37:41.098255 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.098185    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"]
Sep 17 19:37:41.100601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.100584    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.103067 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.103048    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 19:37:41.112006 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.111985    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"]
Sep 17 19:37:41.139486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.139459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.139600 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.139495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d31d3d90-0243-4436-82e9-2db6e4929c74-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.139600 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.139530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpp88\" (UniqueName: \"kubernetes.io/projected/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-api-access-hpp88\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.139709 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.139602    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d31d3d90-0243-4436-82e9-2db6e4929c74-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.139709 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.139686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.139945 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.139927    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"]
Sep 17 19:37:41.145122 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.145096    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-mjpvl"]
Sep 17 19:37:41.240426 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.240567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.240567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.240567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.240730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.240730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.240730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d31d3d90-0243-4436-82e9-2db6e4929c74-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.240730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-api-access-28df8\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.240730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpp88\" (UniqueName: \"kubernetes.io/projected/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-api-access-hpp88\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.241060 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.240757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d31d3d90-0243-4436-82e9-2db6e4929c74-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.241283 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.241257    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d31d3d90-0243-4436-82e9-2db6e4929c74-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.241410 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.241294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.241410 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.241260    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.243748 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.243729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d31d3d90-0243-4436-82e9-2db6e4929c74-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.248813 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.248790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpp88\" (UniqueName: \"kubernetes.io/projected/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-api-access-hpp88\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.315779 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.315755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:41.341497 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.341467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-api-access-28df8\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.341640 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.341513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.341640 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.341533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.341640 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.341559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.341640 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.341587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.341851 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:37:41.341649    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 19:37:41.341851 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:37:41.341719    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-proxy-tls podName:de5edc7a-b01b-4e63-b16e-9f63d241ee8d nodeName:}" failed. No retries permitted until 2026-09-17 19:37:41.841699695 +0000 UTC m=+1880.969441086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-proxy-tls") pod "isvc-canary-rollback-predictor-97977b6b5-9smzx" (UID: "de5edc7a-b01b-4e63-b16e-9f63d241ee8d") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 19:37:41.342378 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.342023    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.342378 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.342313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.342804 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.342660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.352648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.352557    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-api-access-28df8\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.596041 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.596013    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"]
Sep 17 19:37:41.596647 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:37:41.596609    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31d3d90_0243_4436_82e9_2db6e4929c74.slice/crio-6cc9e076278de32ed29a44f2bab9d9a03275ccb83df1ac35b327cd569a76eacd WatchSource:0}: Error finding container 6cc9e076278de32ed29a44f2bab9d9a03275ccb83df1ac35b327cd569a76eacd: Status 404 returned error can't find the container with id 6cc9e076278de32ed29a44f2bab9d9a03275ccb83df1ac35b327cd569a76eacd
Sep 17 19:37:41.816348 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.816305    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" event={"ID":"d31d3d90-0243-4436-82e9-2db6e4929c74","Type":"ContainerStarted","Data":"9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971"}
Sep 17 19:37:41.816841 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.816397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" event={"ID":"d31d3d90-0243-4436-82e9-2db6e4929c74","Type":"ContainerStarted","Data":"6cc9e076278de32ed29a44f2bab9d9a03275ccb83df1ac35b327cd569a76eacd"}
Sep 17 19:37:41.847150 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.847122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:41.849665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:41.849637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-9smzx\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:42.010357 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:42.010325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:42.289461 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:37:42.289427    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5edc7a_b01b_4e63_b16e_9f63d241ee8d.slice/crio-cd7af1e61d85ce2539860d16a079cbecc4c4fb11cfb5a418b6e4a328a839a40d WatchSource:0}: Error finding container cd7af1e61d85ce2539860d16a079cbecc4c4fb11cfb5a418b6e4a328a839a40d: Status 404 returned error can't find the container with id cd7af1e61d85ce2539860d16a079cbecc4c4fb11cfb5a418b6e4a328a839a40d
Sep 17 19:37:42.290392 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:42.290367    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"]
Sep 17 19:37:42.823811 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:42.823772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" event={"ID":"de5edc7a-b01b-4e63-b16e-9f63d241ee8d","Type":"ContainerStarted","Data":"603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb"}
Sep 17 19:37:42.824330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:42.823851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" event={"ID":"de5edc7a-b01b-4e63-b16e-9f63d241ee8d","Type":"ContainerStarted","Data":"cd7af1e61d85ce2539860d16a079cbecc4c4fb11cfb5a418b6e4a328a839a40d"}
Sep 17 19:37:43.039657 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:43.039602    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db1e2f5-96cc-4ca4-9508-e524684d08f7" path="/var/lib/kubelet/pods/8db1e2f5-96cc-4ca4-9508-e524684d08f7/volumes"
Sep 17 19:37:45.838627 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:45.838591    2583 generic.go:356] "Generic (PLEG): container finished" podID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerID="9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971" exitCode=0
Sep 17 19:37:45.839038 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:45.838655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" event={"ID":"d31d3d90-0243-4436-82e9-2db6e4929c74","Type":"ContainerDied","Data":"9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971"}
Sep 17 19:37:46.845014 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:46.844975    2583 generic.go:356] "Generic (PLEG): container finished" podID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerID="603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb" exitCode=0
Sep 17 19:37:46.845492 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:46.845052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" event={"ID":"de5edc7a-b01b-4e63-b16e-9f63d241ee8d","Type":"ContainerDied","Data":"603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb"}
Sep 17 19:37:46.847414 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:46.847389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" event={"ID":"d31d3d90-0243-4436-82e9-2db6e4929c74","Type":"ContainerStarted","Data":"952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1"}
Sep 17 19:37:46.847525 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:46.847444    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" event={"ID":"d31d3d90-0243-4436-82e9-2db6e4929c74","Type":"ContainerStarted","Data":"8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d"}
Sep 17 19:37:46.847691 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:46.847637    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:46.884316 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:46.884270    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podStartSLOduration=2.830711939 podStartE2EDuration="6.88425568s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.053543741s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:37:46.882542894 +0000 UTC m=+1886.010284294" watchObservedRunningTime="2026-09-17 19:37:46.88425568 +0000 UTC m=+1886.011997081"
Sep 17 19:37:47.854574 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:47.854542    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" event={"ID":"de5edc7a-b01b-4e63-b16e-9f63d241ee8d","Type":"ContainerStarted","Data":"f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9"}
Sep 17 19:37:47.855007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:47.854602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" event={"ID":"de5edc7a-b01b-4e63-b16e-9f63d241ee8d","Type":"ContainerStarted","Data":"434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884"}
Sep 17 19:37:47.855007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:47.854806    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:47.855119 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:47.855037    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:47.855119 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:47.855082    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:47.856001 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:47.855977    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:37:47.856086 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:47.855994    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:37:47.875511 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:47.875466    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podStartSLOduration=2.91616041 podStartE2EDuration="6.875453223s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959292813s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:37:47.873877967 +0000 UTC m=+1887.001619372" watchObservedRunningTime="2026-09-17 19:37:47.875453223 +0000 UTC m=+1887.003194674"
Sep 17 19:37:48.789873 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:48.789842    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:37:48.859241 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:48.859200    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:37:48.859686 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:48.859319    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:37:53.864187 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:53.864146    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:37:53.864716 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:53.864678    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:37:53.864842 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:53.864814    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:37:53.865308 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:37:53.865287    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:00.683268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:00.683234    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"]
Sep 17 19:38:00.683765 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:00.683477    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kserve-container" containerID="cri-o://57f7f0b0fe4dddafa8c5e239cd2e1ba88edc856ab262759050653b26c20114c1" gracePeriod=30
Sep 17 19:38:00.683765 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:00.683506    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy" containerID="cri-o://e27cbec1d1fb5c2dcdd1eb8e892a84f15f00956bdaa874351521dce10fb96268" gracePeriod=30
Sep 17 19:38:00.912517 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:00.912479    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerID="e27cbec1d1fb5c2dcdd1eb8e892a84f15f00956bdaa874351521dce10fb96268" exitCode=2
Sep 17 19:38:00.912713 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:00.912547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" event={"ID":"8a5ee734-03f4-46a8-9210-f718142ba5ba","Type":"ContainerDied","Data":"e27cbec1d1fb5c2dcdd1eb8e892a84f15f00956bdaa874351521dce10fb96268"}
Sep 17 19:38:03.778535 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:03.778492    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:38:03.865258 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:03.865222    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:03.865449 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:03.865222    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:05.895592 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.895552    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"]
Sep 17 19:38:05.899325 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.899306    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:05.901646 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.901605    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 19:38:05.901784 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.901676    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:38:05.910852 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.910830    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"]
Sep 17 19:38:05.962081 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.962051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-api-access-46vfn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:05.962264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.962094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:05.962264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.962115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:05.962264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.962196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:05.962264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:05.962226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.063265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.063223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-api-access-46vfn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.063431 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.063275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.063431 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.063307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.063431 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.063347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.063431 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.063373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.063599 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:38:06.063527    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 19:38:06.063671 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:38:06.063644    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-proxy-tls podName:065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f nodeName:}" failed. No retries permitted until 2026-09-17 19:38:06.563583479 +0000 UTC m=+1905.691324858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" (UID: "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 19:38:06.063857 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.063833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.064062 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.064045    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.064107 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.064056    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.080082 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.080049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-api-access-46vfn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.568487 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.568451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.570991 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.570961    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-gpntl\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:06.810290 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:06.810242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:07.083127 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:07.083099    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"]
Sep 17 19:38:07.085041 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:38:07.085013    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065c1bc6_e058_4cb1_a3cb_5ccd82e4c01f.slice/crio-3935d79d821762871aa3d8fb91238a7cc938c7af50c1a5a1d1ad619c882da9cf WatchSource:0}: Error finding container 3935d79d821762871aa3d8fb91238a7cc938c7af50c1a5a1d1ad619c882da9cf: Status 404 returned error can't find the container with id 3935d79d821762871aa3d8fb91238a7cc938c7af50c1a5a1d1ad619c882da9cf
Sep 17 19:38:07.945993 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:07.945954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" event={"ID":"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f","Type":"ContainerStarted","Data":"b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2"}
Sep 17 19:38:07.946207 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:07.946015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" event={"ID":"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f","Type":"ContainerStarted","Data":"3935d79d821762871aa3d8fb91238a7cc938c7af50c1a5a1d1ad619c882da9cf"}
Sep 17 19:38:08.778126 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:08.778082    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:38:11.969265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:11.969182    2583 generic.go:356] "Generic (PLEG): container finished" podID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerID="b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2" exitCode=0
Sep 17 19:38:11.969657 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:11.969263    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" event={"ID":"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f","Type":"ContainerDied","Data":"b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2"}
Sep 17 19:38:12.975918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:12.975881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" event={"ID":"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f","Type":"ContainerStarted","Data":"51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75"}
Sep 17 19:38:12.976334 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:12.975977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" event={"ID":"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f","Type":"ContainerStarted","Data":"a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888"}
Sep 17 19:38:12.976334 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:12.976182    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:12.996988 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:12.996937    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podStartSLOduration=3.434898809 podStartE2EDuration="7.996922402s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.562023593s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:38:12.995843086 +0000 UTC m=+1912.123584478" watchObservedRunningTime="2026-09-17 19:38:12.996922402 +0000 UTC m=+1912.124663803"
Sep 17 19:38:13.778889 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:13.778846    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:38:13.779069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:13.778961    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:38:13.865245 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:13.865208    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:13.865421 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:13.865209    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:13.980479 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:13.980449    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:13.981666 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:13.981641    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:38:14.984881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:14.984835    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:38:18.778700 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:18.778655    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:38:19.989495 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:19.989467    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:19.989893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:19.989802    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 19:38:23.778853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:23.778815    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:38:23.864866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:23.864824    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:23.865198 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:23.865179    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:28.778418 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:28.778375    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 19:38:29.991075 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:29.991043    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:38:31.060715 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.060634    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerID="57f7f0b0fe4dddafa8c5e239cd2e1ba88edc856ab262759050653b26c20114c1" exitCode=137
Sep 17 19:38:31.060715 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.060648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" event={"ID":"8a5ee734-03f4-46a8-9210-f718142ba5ba","Type":"ContainerDied","Data":"57f7f0b0fe4dddafa8c5e239cd2e1ba88edc856ab262759050653b26c20114c1"}
Sep 17 19:38:31.327443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.327418    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:38:31.491720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.491686    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5ee734-03f4-46a8-9210-f718142ba5ba-kserve-provision-location\") pod \"8a5ee734-03f4-46a8-9210-f718142ba5ba\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") "
Sep 17 19:38:31.491904 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.491802    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fltbc\" (UniqueName: \"kubernetes.io/projected/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-api-access-fltbc\") pod \"8a5ee734-03f4-46a8-9210-f718142ba5ba\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") "
Sep 17 19:38:31.491904 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.491848    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-cabundle-cert\") pod \"8a5ee734-03f4-46a8-9210-f718142ba5ba\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") "
Sep 17 19:38:31.491904 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.491871    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-rbac-proxy-sar-config\") pod \"8a5ee734-03f4-46a8-9210-f718142ba5ba\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") "
Sep 17 19:38:31.492077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.492023    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a5ee734-03f4-46a8-9210-f718142ba5ba-proxy-tls\") pod \"8a5ee734-03f4-46a8-9210-f718142ba5ba\" (UID: \"8a5ee734-03f4-46a8-9210-f718142ba5ba\") "
Sep 17 19:38:31.492246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.492214    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a5ee734-03f4-46a8-9210-f718142ba5ba" (UID: "8a5ee734-03f4-46a8-9210-f718142ba5ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:38:31.492246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.492225    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a5ee734-03f4-46a8-9210-f718142ba5ba" (UID: "8a5ee734-03f4-46a8-9210-f718142ba5ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:38:31.492417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.492345    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:38:31.492417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.492363    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:38:31.494159 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.494127    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5ee734-03f4-46a8-9210-f718142ba5ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a5ee734-03f4-46a8-9210-f718142ba5ba" (UID: "8a5ee734-03f4-46a8-9210-f718142ba5ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:38:31.494159 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.494133    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-api-access-fltbc" (OuterVolumeSpecName: "kube-api-access-fltbc") pod "8a5ee734-03f4-46a8-9210-f718142ba5ba" (UID: "8a5ee734-03f4-46a8-9210-f718142ba5ba"). InnerVolumeSpecName "kube-api-access-fltbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:38:31.503062 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.503038    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5ee734-03f4-46a8-9210-f718142ba5ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a5ee734-03f4-46a8-9210-f718142ba5ba" (UID: "8a5ee734-03f4-46a8-9210-f718142ba5ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:38:31.593417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.593349    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a5ee734-03f4-46a8-9210-f718142ba5ba-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:38:31.593417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.593374    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a5ee734-03f4-46a8-9210-f718142ba5ba-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:38:31.593417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:31.593384    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fltbc\" (UniqueName: \"kubernetes.io/projected/8a5ee734-03f4-46a8-9210-f718142ba5ba-kube-api-access-fltbc\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:38:32.066825 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:32.066787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq" event={"ID":"8a5ee734-03f4-46a8-9210-f718142ba5ba","Type":"ContainerDied","Data":"30ed1a5dfb960cca95e563dcbcf3e8383a58dc12614703e073fd86be8e967a90"}
Sep 17 19:38:32.067230 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:32.066843    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"
Sep 17 19:38:32.067230 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:32.066871    2583 scope.go:117] "RemoveContainer" containerID="e27cbec1d1fb5c2dcdd1eb8e892a84f15f00956bdaa874351521dce10fb96268"
Sep 17 19:38:32.075650 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:32.075631    2583 scope.go:117] "RemoveContainer" containerID="57f7f0b0fe4dddafa8c5e239cd2e1ba88edc856ab262759050653b26c20114c1"
Sep 17 19:38:32.083657 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:32.083637    2583 scope.go:117] "RemoveContainer" containerID="72440f2f58e994bf6487da71f1a58b9caedd6d73baf393fc036107eedd02f90b"
Sep 17 19:38:32.091559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:32.091534    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"]
Sep 17 19:38:32.097919 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:32.097896    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-wtmtq"]
Sep 17 19:38:33.039141 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:33.039110    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" path="/var/lib/kubelet/pods/8a5ee734-03f4-46a8-9210-f718142ba5ba/volumes"
Sep 17 19:38:33.865590 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:33.865552    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:33.866008 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:33.865557    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:43.865181 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:43.865142    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:43.865752 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:43.865181    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:47.476807 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:47.476708    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"]
Sep 17 19:38:47.477291 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:47.476956    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kserve-container" containerID="cri-o://a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888" gracePeriod=30
Sep 17 19:38:47.477291 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:47.477027    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy" containerID="cri-o://51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75" gracePeriod=30
Sep 17 19:38:48.143859 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:48.143824    2583 generic.go:356] "Generic (PLEG): container finished" podID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerID="51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75" exitCode=2
Sep 17 19:38:48.144042 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:48.143912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" event={"ID":"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f","Type":"ContainerDied","Data":"51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75"}
Sep 17 19:38:49.985988 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:49.985944    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:38:52.670758 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.670724    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"]
Sep 17 19:38:52.671754 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.671704    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy"
Sep 17 19:38:52.671754 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.671754    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy"
Sep 17 19:38:52.671954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.671789    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="storage-initializer"
Sep 17 19:38:52.671954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.671806    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="storage-initializer"
Sep 17 19:38:52.671954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.671856    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kserve-container"
Sep 17 19:38:52.671954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.671865    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kserve-container"
Sep 17 19:38:52.672136 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.672090    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kserve-container"
Sep 17 19:38:52.672136 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.672107    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5ee734-03f4-46a8-9210-f718142ba5ba" containerName="kube-rbac-proxy"
Sep 17 19:38:52.678593 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.678568    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.681003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.680984    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 19:38:52.681104 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.681034    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 19:38:52.684553 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.684532    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"]
Sep 17 19:38:52.774815 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.774784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.774976 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.774827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.774976 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.774887    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv72d\" (UniqueName: \"kubernetes.io/projected/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-api-access-sv72d\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.774976 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.774946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead24ff9-e971-4443-940b-e1cd9afdbdbf-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.775123 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.774999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.876317 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.876285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead24ff9-e971-4443-940b-e1cd9afdbdbf-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.876489 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.876361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.876489 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.876392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.876489 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.876426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.876489 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.876472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv72d\" (UniqueName: \"kubernetes.io/projected/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-api-access-sv72d\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.876920 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.876897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.877140 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.877122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.877186 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.877138    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.879019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.878999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead24ff9-e971-4443-940b-e1cd9afdbdbf-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.884401 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.884379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv72d\" (UniqueName: \"kubernetes.io/projected/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-api-access-sv72d\") pod \"isvc-triton-predictor-5d7db58676-stxwp\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:52.990000 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:52.989920    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:38:53.266211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:53.266187    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"]
Sep 17 19:38:53.268340 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:38:53.268311    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead24ff9_e971_4443_940b_e1cd9afdbdbf.slice/crio-9029238171eea27760e96b39e897e1b511608c5fcae5b0fac9a58dea12fb46e1 WatchSource:0}: Error finding container 9029238171eea27760e96b39e897e1b511608c5fcae5b0fac9a58dea12fb46e1: Status 404 returned error can't find the container with id 9029238171eea27760e96b39e897e1b511608c5fcae5b0fac9a58dea12fb46e1
Sep 17 19:38:53.865038 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:53.864986    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:53.865528 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:53.865216    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:54.172131 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:54.172096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" event={"ID":"ead24ff9-e971-4443-940b-e1cd9afdbdbf","Type":"ContainerStarted","Data":"68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec"}
Sep 17 19:38:54.172337 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:54.172156    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" event={"ID":"ead24ff9-e971-4443-940b-e1cd9afdbdbf","Type":"ContainerStarted","Data":"9029238171eea27760e96b39e897e1b511608c5fcae5b0fac9a58dea12fb46e1"}
Sep 17 19:38:54.985778 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:54.985732    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:38:58.191985 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:58.191950    2583 generic.go:356] "Generic (PLEG): container finished" podID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerID="68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec" exitCode=0
Sep 17 19:38:58.192388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:58.192006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" event={"ID":"ead24ff9-e971-4443-940b-e1cd9afdbdbf","Type":"ContainerDied","Data":"68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec"}
Sep 17 19:38:59.986410 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:59.985717    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:38:59.986410 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:38:59.985891    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:39:03.038543 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:03.037901    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:39:03.866099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:03.865719    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:39:04.985993 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:04.985605    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:39:09.985475 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:09.985423    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:39:12.035184 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:12.035147    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:39:12.175102 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:12.175060    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"]
Sep 17 19:39:12.176108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:12.175448    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container" containerID="cri-o://434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884" gracePeriod=30
Sep 17 19:39:12.176108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:12.175738    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kube-rbac-proxy" containerID="cri-o://f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9" gracePeriod=30
Sep 17 19:39:12.176908 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:12.176401    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:39:12.320184 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:12.319765    2583 generic.go:356] "Generic (PLEG): container finished" podID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerID="f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9" exitCode=2
Sep 17 19:39:12.320184 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:12.320070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" event={"ID":"de5edc7a-b01b-4e63-b16e-9f63d241ee8d","Type":"ContainerDied","Data":"f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9"}
Sep 17 19:39:13.860121 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:13.860077    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 17 19:39:14.986149 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:14.986088    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 19:39:18.237399 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.237244    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:39:18.249298 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.249250    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:39:18.336437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336396    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-rbac-proxy-sar-config\") pod \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") "
Sep 17 19:39:18.336652 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-api-access-28df8\") pod \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") "
Sep 17 19:39:18.336652 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336529    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-proxy-tls\") pod \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") "
Sep 17 19:39:18.336652 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-rbac-proxy-sar-config\") pod \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") "
Sep 17 19:39:18.336652 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336599    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-proxy-tls\") pod \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") "
Sep 17 19:39:18.336898 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336662    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kserve-provision-location\") pod \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") "
Sep 17 19:39:18.336898 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336692    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kserve-provision-location\") pod \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") "
Sep 17 19:39:18.336898 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-cabundle-cert\") pod \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\" (UID: \"de5edc7a-b01b-4e63-b16e-9f63d241ee8d\") "
Sep 17 19:39:18.336898 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336749    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-cabundle-cert\") pod \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") "
Sep 17 19:39:18.336898 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.336822    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-api-access-46vfn\") pod \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\" (UID: \"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f\") "
Sep 17 19:39:18.338191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.337522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de5edc7a-b01b-4e63-b16e-9f63d241ee8d" (UID: "de5edc7a-b01b-4e63-b16e-9f63d241ee8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:39:18.338191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.337828    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de5edc7a-b01b-4e63-b16e-9f63d241ee8d" (UID: "de5edc7a-b01b-4e63-b16e-9f63d241ee8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:18.338191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.338117    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" (UID: "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:39:18.338191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.338164    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" (UID: "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:39:18.338569 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.338455    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de5edc7a-b01b-4e63-b16e-9f63d241ee8d" (UID: "de5edc7a-b01b-4e63-b16e-9f63d241ee8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:39:18.339797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.339761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de5edc7a-b01b-4e63-b16e-9f63d241ee8d" (UID: "de5edc7a-b01b-4e63-b16e-9f63d241ee8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:39:18.339918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.339828    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-api-access-46vfn" (OuterVolumeSpecName: "kube-api-access-46vfn") pod "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" (UID: "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f"). InnerVolumeSpecName "kube-api-access-46vfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:39:18.340734 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.340708    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-api-access-28df8" (OuterVolumeSpecName: "kube-api-access-28df8") pod "de5edc7a-b01b-4e63-b16e-9f63d241ee8d" (UID: "de5edc7a-b01b-4e63-b16e-9f63d241ee8d"). InnerVolumeSpecName "kube-api-access-28df8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:39:18.342446 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.342420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" (UID: "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:39:18.349538 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.349498    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" (UID: "065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:18.361810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.361780    2583 generic.go:356] "Generic (PLEG): container finished" podID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerID="434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884" exitCode=0
Sep 17 19:39:18.361946 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.361918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" event={"ID":"de5edc7a-b01b-4e63-b16e-9f63d241ee8d","Type":"ContainerDied","Data":"434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884"}
Sep 17 19:39:18.362010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.361975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx" event={"ID":"de5edc7a-b01b-4e63-b16e-9f63d241ee8d","Type":"ContainerDied","Data":"cd7af1e61d85ce2539860d16a079cbecc4c4fb11cfb5a418b6e4a328a839a40d"}
Sep 17 19:39:18.362071 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.362011    2583 scope.go:117] "RemoveContainer" containerID="f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9"
Sep 17 19:39:18.362235 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.362210    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"
Sep 17 19:39:18.368524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.368481    2583 generic.go:356] "Generic (PLEG): container finished" podID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerID="a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888" exitCode=137
Sep 17 19:39:18.368657 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.368565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" event={"ID":"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f","Type":"ContainerDied","Data":"a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888"}
Sep 17 19:39:18.368657 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.368632    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl" event={"ID":"065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f","Type":"ContainerDied","Data":"3935d79d821762871aa3d8fb91238a7cc938c7af50c1a5a1d1ad619c882da9cf"}
Sep 17 19:39:18.368767 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.368744    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"
Sep 17 19:39:18.378033 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.378013    2583 scope.go:117] "RemoveContainer" containerID="434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884"
Sep 17 19:39:18.389561 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.389515    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"]
Sep 17 19:39:18.389949 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.389918    2583 scope.go:117] "RemoveContainer" containerID="603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb"
Sep 17 19:39:18.393297 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.393250    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-9smzx"]
Sep 17 19:39:18.401204 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.401185    2583 scope.go:117] "RemoveContainer" containerID="f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9"
Sep 17 19:39:18.401752 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:39:18.401707    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9\": container with ID starting with f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9 not found: ID does not exist" containerID="f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9"
Sep 17 19:39:18.401752 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.401762    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9"} err="failed to get container status \"f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9\": rpc error: code = NotFound desc = could not find container \"f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9\": container with ID starting with f990a42b83372fab4b7281a00fe18a558ecba83c366ffb6b7beedd7e5a2601d9 not found: ID does not exist"
Sep 17 19:39:18.401752 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.401781    2583 scope.go:117] "RemoveContainer" containerID="434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884"
Sep 17 19:39:18.402729 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:39:18.402093    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884\": container with ID starting with 434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884 not found: ID does not exist" containerID="434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884"
Sep 17 19:39:18.402729 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.402123    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884"} err="failed to get container status \"434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884\": rpc error: code = NotFound desc = could not find container \"434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884\": container with ID starting with 434e37b1d6e49f2e42c44e5195d2037a295690652596c63bdf8fe3e998425884 not found: ID does not exist"
Sep 17 19:39:18.402729 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.402154    2583 scope.go:117] "RemoveContainer" containerID="603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb"
Sep 17 19:39:18.402729 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:39:18.402463    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb\": container with ID starting with 603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb not found: ID does not exist" containerID="603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb"
Sep 17 19:39:18.402729 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.402502    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb"} err="failed to get container status \"603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb\": rpc error: code = NotFound desc = could not find container \"603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb\": container with ID starting with 603f94e245008b5fa85fe3ed7b7d6f291492d92fe77cf301bc1eb855db8337bb not found: ID does not exist"
Sep 17 19:39:18.402729 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.402524    2583 scope.go:117] "RemoveContainer" containerID="51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75"
Sep 17 19:39:18.405593 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.405572    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"]
Sep 17 19:39:18.412375 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.412351    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-gpntl"]
Sep 17 19:39:18.414104 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.414084    2583 scope.go:117] "RemoveContainer" containerID="a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888"
Sep 17 19:39:18.427891 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.427870    2583 scope.go:117] "RemoveContainer" containerID="b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2"
Sep 17 19:39:18.437783 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.437764    2583 scope.go:117] "RemoveContainer" containerID="51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75"
Sep 17 19:39:18.437961 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.437938    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.438040 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.437967    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kube-api-access-28df8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.438040 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.437984    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.438040 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438002    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.438040 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438015    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.438040 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438028    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.438305 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438042    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.438305 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:39:18.438047    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75\": container with ID starting with 51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75 not found: ID does not exist" containerID="51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75"
Sep 17 19:39:18.438305 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438079    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75"} err="failed to get container status \"51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75\": rpc error: code = NotFound desc = could not find container \"51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75\": container with ID starting with 51b0a38ef38e8b7e5816dbb38a3232fa17570cfb0975f2bf11fd5ebbba54eb75 not found: ID does not exist"
Sep 17 19:39:18.438305 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438104    2583 scope.go:117] "RemoveContainer" containerID="a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888"
Sep 17 19:39:18.438305 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438055    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de5edc7a-b01b-4e63-b16e-9f63d241ee8d-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.438305 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438159    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.438305 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438172    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46vfn\" (UniqueName: \"kubernetes.io/projected/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f-kube-api-access-46vfn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:18.439089 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:39:18.438494    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888\": container with ID starting with a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888 not found: ID does not exist" containerID="a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888"
Sep 17 19:39:18.439089 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438523    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888"} err="failed to get container status \"a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888\": rpc error: code = NotFound desc = could not find container \"a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888\": container with ID starting with a1278dc8aab11bafcf99339a1abaf2585a032fafc7cc84ad347e4f1bf0c55888 not found: ID does not exist"
Sep 17 19:39:18.439089 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438544    2583 scope.go:117] "RemoveContainer" containerID="b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2"
Sep 17 19:39:18.439089 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:39:18.438857    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2\": container with ID starting with b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2 not found: ID does not exist" containerID="b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2"
Sep 17 19:39:18.439089 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:18.438882    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2"} err="failed to get container status \"b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2\": rpc error: code = NotFound desc = could not find container \"b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2\": container with ID starting with b35d31a0da7a6f14f47454ed2d39e0fad2caa1d299035206e7c7b6afe82af2e2 not found: ID does not exist"
Sep 17 19:39:19.044010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:19.043977    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" path="/var/lib/kubelet/pods/065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f/volumes"
Sep 17 19:39:19.044701 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:19.044673    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" path="/var/lib/kubelet/pods/de5edc7a-b01b-4e63-b16e-9f63d241ee8d/volumes"
Sep 17 19:39:20.947722 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.947685    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"]
Sep 17 19:39:20.948176 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.947995    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" containerID="cri-o://8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d" gracePeriod=30
Sep 17 19:39:20.948262 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.948181    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kube-rbac-proxy" containerID="cri-o://952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1" gracePeriod=30
Sep 17 19:39:20.992358 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.991974    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"]
Sep 17 19:39:20.992570 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992547    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kube-rbac-proxy"
Sep 17 19:39:20.992659 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992571    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kube-rbac-proxy"
Sep 17 19:39:20.992659 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992592    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy"
Sep 17 19:39:20.992659 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992600    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy"
Sep 17 19:39:20.992659 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992630    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container"
Sep 17 19:39:20.992659 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992639    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992663    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="storage-initializer"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992672    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="storage-initializer"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992681    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kserve-container"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992689    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kserve-container"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992778    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="storage-initializer"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992786    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="storage-initializer"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992874    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kserve-container"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992886    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5edc7a-b01b-4e63-b16e-9f63d241ee8d" containerName="kube-rbac-proxy"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992895    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kube-rbac-proxy"
Sep 17 19:39:20.992915 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:20.992908    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="065c1bc6-e058-4cb1-a3cb-5ccd82e4c01f" containerName="kserve-container"
Sep 17 19:39:21.025025 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.024790    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"]
Sep 17 19:39:21.025025 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.024933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.027444 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.027414    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 19:39:21.062809 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.062780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.063233 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.063197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.063605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.063580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.063749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.063691    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qktrz\" (UniqueName: \"kubernetes.io/projected/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-api-access-qktrz\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.063749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.063742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.164732 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.164698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.164934 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.164774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.164934 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.164822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.164934 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.164862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qktrz\" (UniqueName: \"kubernetes.io/projected/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-api-access-qktrz\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.164934 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.164896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.165390 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.165336    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.165807 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.165757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.165807 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.165779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.168726 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.168700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.175842 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.175800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktrz\" (UniqueName: \"kubernetes.io/projected/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-api-access-qktrz\") pod \"isvc-canary-rollback-predictor-97977b6b5-lr59l\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.340282 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.340193    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:21.395003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.394803    2583 generic.go:356] "Generic (PLEG): container finished" podID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerID="952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1" exitCode=2
Sep 17 19:39:21.395003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.394891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" event={"ID":"d31d3d90-0243-4436-82e9-2db6e4929c74","Type":"ContainerDied","Data":"952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1"}
Sep 17 19:39:21.656875 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:21.656790    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"]
Sep 17 19:39:21.658775 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:39:21.658741    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a62bdf6_b45b_4a36_940b_d618fbab5e8c.slice/crio-795d1244d8e8f3bfda16fb91f744fe86c7b1c346c3cc512747957e1624c492e2 WatchSource:0}: Error finding container 795d1244d8e8f3bfda16fb91f744fe86c7b1c346c3cc512747957e1624c492e2: Status 404 returned error can't find the container with id 795d1244d8e8f3bfda16fb91f744fe86c7b1c346c3cc512747957e1624c492e2
Sep 17 19:39:22.034754 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:22.034701    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:39:22.403567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:22.403477    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" event={"ID":"5a62bdf6-b45b-4a36-940b-d618fbab5e8c","Type":"ContainerStarted","Data":"9aa931845810808d9c2af18f35da30a4899008107693ef11cf1b37fc52bed68e"}
Sep 17 19:39:22.403567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:22.403554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" event={"ID":"5a62bdf6-b45b-4a36-940b-d618fbab5e8c","Type":"ContainerStarted","Data":"795d1244d8e8f3bfda16fb91f744fe86c7b1c346c3cc512747957e1624c492e2"}
Sep 17 19:39:23.859545 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:23.859495    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 19:39:26.428564 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.428521    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerID="9aa931845810808d9c2af18f35da30a4899008107693ef11cf1b37fc52bed68e" exitCode=0
Sep 17 19:39:26.428999 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.428592    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" event={"ID":"5a62bdf6-b45b-4a36-940b-d618fbab5e8c","Type":"ContainerDied","Data":"9aa931845810808d9c2af18f35da30a4899008107693ef11cf1b37fc52bed68e"}
Sep 17 19:39:26.857167 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.857096    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:39:26.920707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.920038    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-rbac-proxy-sar-config\") pod \"d31d3d90-0243-4436-82e9-2db6e4929c74\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") "
Sep 17 19:39:26.920707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.920101    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d31d3d90-0243-4436-82e9-2db6e4929c74-proxy-tls\") pod \"d31d3d90-0243-4436-82e9-2db6e4929c74\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") "
Sep 17 19:39:26.920707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.920121    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-cabundle-cert\") pod \"d31d3d90-0243-4436-82e9-2db6e4929c74\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") "
Sep 17 19:39:26.920707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.920172    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d31d3d90-0243-4436-82e9-2db6e4929c74-kserve-provision-location\") pod \"d31d3d90-0243-4436-82e9-2db6e4929c74\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") "
Sep 17 19:39:26.920707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.920223    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpp88\" (UniqueName: \"kubernetes.io/projected/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-api-access-hpp88\") pod \"d31d3d90-0243-4436-82e9-2db6e4929c74\" (UID: \"d31d3d90-0243-4436-82e9-2db6e4929c74\") "
Sep 17 19:39:26.921080 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.920780    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31d3d90-0243-4436-82e9-2db6e4929c74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d31d3d90-0243-4436-82e9-2db6e4929c74" (UID: "d31d3d90-0243-4436-82e9-2db6e4929c74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:26.921080 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.920944    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d31d3d90-0243-4436-82e9-2db6e4929c74" (UID: "d31d3d90-0243-4436-82e9-2db6e4929c74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:39:26.921080 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.920986    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d31d3d90-0243-4436-82e9-2db6e4929c74" (UID: "d31d3d90-0243-4436-82e9-2db6e4929c74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:39:26.923462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.923434    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31d3d90-0243-4436-82e9-2db6e4929c74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d31d3d90-0243-4436-82e9-2db6e4929c74" (UID: "d31d3d90-0243-4436-82e9-2db6e4929c74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:39:26.923858 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:26.923829    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-api-access-hpp88" (OuterVolumeSpecName: "kube-api-access-hpp88") pod "d31d3d90-0243-4436-82e9-2db6e4929c74" (UID: "d31d3d90-0243-4436-82e9-2db6e4929c74"). InnerVolumeSpecName "kube-api-access-hpp88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:39:27.021462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.021375    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d31d3d90-0243-4436-82e9-2db6e4929c74-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:27.021462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.021413    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpp88\" (UniqueName: \"kubernetes.io/projected/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-api-access-hpp88\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:27.021462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.021428    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:27.021462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.021443    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d31d3d90-0243-4436-82e9-2db6e4929c74-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:27.021462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.021457    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d31d3d90-0243-4436-82e9-2db6e4929c74-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:39:27.437944 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.437909    2583 generic.go:356] "Generic (PLEG): container finished" podID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerID="8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d" exitCode=0
Sep 17 19:39:27.438485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.438121    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"
Sep 17 19:39:27.438485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.438223    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" event={"ID":"d31d3d90-0243-4436-82e9-2db6e4929c74","Type":"ContainerDied","Data":"8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d"}
Sep 17 19:39:27.438485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.438300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f" event={"ID":"d31d3d90-0243-4436-82e9-2db6e4929c74","Type":"ContainerDied","Data":"6cc9e076278de32ed29a44f2bab9d9a03275ccb83df1ac35b327cd569a76eacd"}
Sep 17 19:39:27.438485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.438324    2583 scope.go:117] "RemoveContainer" containerID="952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1"
Sep 17 19:39:27.441189 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.441159    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" event={"ID":"5a62bdf6-b45b-4a36-940b-d618fbab5e8c","Type":"ContainerStarted","Data":"80da9ac30445f50140d6ed2f473b1c572c76ce3f1c811b598af9392df2a49a91"}
Sep 17 19:39:27.441313 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.441218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" event={"ID":"5a62bdf6-b45b-4a36-940b-d618fbab5e8c","Type":"ContainerStarted","Data":"c99130512aba82986f3533c758eebff7764de75f7359fc627a1dd00bb134e605"}
Sep 17 19:39:27.441497 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.441477    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:27.441581 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.441543    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:27.443650 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.443128    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:39:27.452188 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.452158    2583 scope.go:117] "RemoveContainer" containerID="8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d"
Sep 17 19:39:27.461724 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.461635    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"]
Sep 17 19:39:27.463645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.463336    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-kf84f"]
Sep 17 19:39:27.466847 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.466825    2583 scope.go:117] "RemoveContainer" containerID="9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971"
Sep 17 19:39:27.478713 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.478692    2583 scope.go:117] "RemoveContainer" containerID="952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1"
Sep 17 19:39:27.479181 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.479138    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podStartSLOduration=3.22367909 podStartE2EDuration="7.479121678s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.255442588s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:39:27.478414506 +0000 UTC m=+1986.606155907" watchObservedRunningTime="2026-09-17 19:39:27.479121678 +0000 UTC m=+1986.606863079"
Sep 17 19:39:27.479540 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:39:27.479513    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1\": container with ID starting with 952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1 not found: ID does not exist" containerID="952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1"
Sep 17 19:39:27.479863 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.479813    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1"} err="failed to get container status \"952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1\": rpc error: code = NotFound desc = could not find container \"952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1\": container with ID starting with 952c05cb795d158d225bd40157ea2f60a7fd66cb244ce4cb1751851cf057e0a1 not found: ID does not exist"
Sep 17 19:39:27.479963 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.479894    2583 scope.go:117] "RemoveContainer" containerID="8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d"
Sep 17 19:39:27.480761 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:39:27.480728    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d\": container with ID starting with 8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d not found: ID does not exist" containerID="8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d"
Sep 17 19:39:27.480850 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.480763    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d"} err="failed to get container status \"8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d\": rpc error: code = NotFound desc = could not find container \"8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d\": container with ID starting with 8619f95cf0203226cffb8aa587ee0258e3edee2ed281c087a953e62b5d5d317d not found: ID does not exist"
Sep 17 19:39:27.480850 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.480785    2583 scope.go:117] "RemoveContainer" containerID="9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971"
Sep 17 19:39:27.481099 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:39:27.481077    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971\": container with ID starting with 9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971 not found: ID does not exist" containerID="9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971"
Sep 17 19:39:27.481182 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:27.481106    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971"} err="failed to get container status \"9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971\": rpc error: code = NotFound desc = could not find container \"9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971\": container with ID starting with 9a5d0396b7f97b0f3d0ad83b60142b20abce9b28a6a3a07902fb80461b1db971 not found: ID does not exist"
Sep 17 19:39:28.450664 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:28.450457    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:39:29.040970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:29.040940    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" path="/var/lib/kubelet/pods/d31d3d90-0243-4436-82e9-2db6e4929c74/volumes"
Sep 17 19:39:33.456864 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:33.456830    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:39:33.457492 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:33.457455    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:39:43.457789 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:43.457749    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:39:53.457719 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:39:53.457677    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:40:03.457971 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:03.457930    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:40:13.458362 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:13.458314    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:40:23.458258 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:23.458209    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:40:33.458206 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:33.458112    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:40:43.458315 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:43.458268    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:40:53.458003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:53.457962    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:40:55.901069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:55.901031    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" event={"ID":"ead24ff9-e971-4443-940b-e1cd9afdbdbf","Type":"ContainerStarted","Data":"6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd"}
Sep 17 19:40:55.901440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:55.901091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" event={"ID":"ead24ff9-e971-4443-940b-e1cd9afdbdbf","Type":"ContainerStarted","Data":"781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced"}
Sep 17 19:40:55.901440 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:55.901209    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:40:55.921027 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:55.920986    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" podStartSLOduration=2.976132836 podStartE2EDuration="2m3.920972413s" totalImagesPullingTime="1m56.87033024s" totalInitContainerRuntime="4.074509337s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:40:55.918991823 +0000 UTC m=+2075.046733223" watchObservedRunningTime="2026-09-17 19:40:55.920972413 +0000 UTC m=+2075.048713813"
Sep 17 19:40:56.905479 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:56.905445    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:40:56.906692 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:56.906661    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:40:57.033662 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:57.033603    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 19:40:57.909951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:40:57.909912    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 19:41:02.915650 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:02.915596    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:41:02.916413 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:02.916391    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:41:07.038315 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:07.038288    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:41:11.124816 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:11.124784    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"]
Sep 17 19:41:11.125340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:11.125010    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container" containerID="cri-o://c99130512aba82986f3533c758eebff7764de75f7359fc627a1dd00bb134e605" gracePeriod=30
Sep 17 19:41:11.125340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:11.125044    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kube-rbac-proxy" containerID="cri-o://80da9ac30445f50140d6ed2f473b1c572c76ce3f1c811b598af9392df2a49a91" gracePeriod=30
Sep 17 19:41:11.975990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:11.975953    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerID="80da9ac30445f50140d6ed2f473b1c572c76ce3f1c811b598af9392df2a49a91" exitCode=2
Sep 17 19:41:11.976172 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:11.976030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" event={"ID":"5a62bdf6-b45b-4a36-940b-d618fbab5e8c","Type":"ContainerDied","Data":"80da9ac30445f50140d6ed2f473b1c572c76ce3f1c811b598af9392df2a49a91"}
Sep 17 19:41:13.451801 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:13.451757    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 17 19:41:15.994858 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:15.994794    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerID="c99130512aba82986f3533c758eebff7764de75f7359fc627a1dd00bb134e605" exitCode=0
Sep 17 19:41:15.995180 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:15.994867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" event={"ID":"5a62bdf6-b45b-4a36-940b-d618fbab5e8c","Type":"ContainerDied","Data":"c99130512aba82986f3533c758eebff7764de75f7359fc627a1dd00bb134e605"}
Sep 17 19:41:15.995180 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:15.994924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l" event={"ID":"5a62bdf6-b45b-4a36-940b-d618fbab5e8c","Type":"ContainerDied","Data":"795d1244d8e8f3bfda16fb91f744fe86c7b1c346c3cc512747957e1624c492e2"}
Sep 17 19:41:15.995180 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:15.994946    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="795d1244d8e8f3bfda16fb91f744fe86c7b1c346c3cc512747957e1624c492e2"
Sep 17 19:41:16.004930 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.004912    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:41:16.129398 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.129363    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qktrz\" (UniqueName: \"kubernetes.io/projected/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-api-access-qktrz\") pod \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") "
Sep 17 19:41:16.129608 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.129410    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-proxy-tls\") pod \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") "
Sep 17 19:41:16.129608 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.129442    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kserve-provision-location\") pod \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") "
Sep 17 19:41:16.129608 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.129492    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-rbac-proxy-sar-config\") pod \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") "
Sep 17 19:41:16.129608 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.129511    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-cabundle-cert\") pod \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\" (UID: \"5a62bdf6-b45b-4a36-940b-d618fbab5e8c\") "
Sep 17 19:41:16.129893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.129854    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a62bdf6-b45b-4a36-940b-d618fbab5e8c" (UID: "5a62bdf6-b45b-4a36-940b-d618fbab5e8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:16.129893 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.129874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a62bdf6-b45b-4a36-940b-d618fbab5e8c" (UID: "5a62bdf6-b45b-4a36-940b-d618fbab5e8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:41:16.131829 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.131804    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a62bdf6-b45b-4a36-940b-d618fbab5e8c" (UID: "5a62bdf6-b45b-4a36-940b-d618fbab5e8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:41:16.131938 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.131874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-api-access-qktrz" (OuterVolumeSpecName: "kube-api-access-qktrz") pod "5a62bdf6-b45b-4a36-940b-d618fbab5e8c" (UID: "5a62bdf6-b45b-4a36-940b-d618fbab5e8c"). InnerVolumeSpecName "kube-api-access-qktrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:41:16.134311 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.134275    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a62bdf6-b45b-4a36-940b-d618fbab5e8c" (UID: "5a62bdf6-b45b-4a36-940b-d618fbab5e8c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:41:16.192902 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.192867    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"]
Sep 17 19:41:16.193234 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.193136    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kserve-container" containerID="cri-o://781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced" gracePeriod=30
Sep 17 19:41:16.193234 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.193165    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kube-rbac-proxy" containerID="cri-o://6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd" gracePeriod=30
Sep 17 19:41:16.231118 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.231086    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qktrz\" (UniqueName: \"kubernetes.io/projected/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-api-access-qktrz\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:16.231118 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.231121    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:16.231291 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.231135    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:16.231291 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.231151    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:16.231291 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.231165    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a62bdf6-b45b-4a36-940b-d618fbab5e8c-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:16.312580 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312510    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"]
Sep 17 19:41:16.312858 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312844    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kube-rbac-proxy"
Sep 17 19:41:16.312858 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312858    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kube-rbac-proxy"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312875    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312881    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312890    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312896    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312902    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="storage-initializer"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312908    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="storage-initializer"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312917    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kube-rbac-proxy"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312923    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kube-rbac-proxy"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312932    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="storage-initializer"
Sep 17 19:41:16.312953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312937    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="storage-initializer"
Sep 17 19:41:16.313287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312988    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kube-rbac-proxy"
Sep 17 19:41:16.313287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.312995    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" containerName="kserve-container"
Sep 17 19:41:16.313287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.313003    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kube-rbac-proxy"
Sep 17 19:41:16.313287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.313010    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31d3d90-0243-4436-82e9-2db6e4929c74" containerName="kserve-container"
Sep 17 19:41:16.323394 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.323374    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.325932 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.325910    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 19:41:16.325932 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.325926    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 19:41:16.329124 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.329100    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"]
Sep 17 19:41:16.432688 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.432651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.432883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.432702    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.432883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.432757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f0bde55-a4c4-4831-8cde-0b151cc2d864-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.432883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.432818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcgvf\" (UniqueName: \"kubernetes.io/projected/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-api-access-gcgvf\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.432883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.432853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.533465 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.533428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.533465 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.533466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.533777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.533504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f0bde55-a4c4-4831-8cde-0b151cc2d864-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.533777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.533528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcgvf\" (UniqueName: \"kubernetes.io/projected/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-api-access-gcgvf\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.533777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.533546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.533982 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.533958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.534273 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.534246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.534331 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.534261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.536176 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.536155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f0bde55-a4c4-4831-8cde-0b151cc2d864-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.542073 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.542050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcgvf\" (UniqueName: \"kubernetes.io/projected/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-api-access-gcgvf\") pod \"isvc-canary-stop-predictor-55656fdf77-4h2dq\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.634528 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.634455    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:16.921455 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:16.921431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"]
Sep 17 19:41:16.923546 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:41:16.923519    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0bde55_a4c4_4831_8cde_0b151cc2d864.slice/crio-ac99229eb8abf7978f2b865389921a3c3ff332d7ea62443380f8ecbf5999a5ff WatchSource:0}: Error finding container ac99229eb8abf7978f2b865389921a3c3ff332d7ea62443380f8ecbf5999a5ff: Status 404 returned error can't find the container with id ac99229eb8abf7978f2b865389921a3c3ff332d7ea62443380f8ecbf5999a5ff
Sep 17 19:41:17.000883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:17.000851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" event={"ID":"8f0bde55-a4c4-4831-8cde-0b151cc2d864","Type":"ContainerStarted","Data":"ac99229eb8abf7978f2b865389921a3c3ff332d7ea62443380f8ecbf5999a5ff"}
Sep 17 19:41:17.003265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:17.003241    2583 generic.go:356] "Generic (PLEG): container finished" podID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerID="6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd" exitCode=2
Sep 17 19:41:17.003362 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:17.003313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" event={"ID":"ead24ff9-e971-4443-940b-e1cd9afdbdbf","Type":"ContainerDied","Data":"6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd"}
Sep 17 19:41:17.003362 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:17.003352    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"
Sep 17 19:41:17.028177 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:17.028153    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"]
Sep 17 19:41:17.044553 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:17.044524    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-lr59l"]
Sep 17 19:41:17.910672 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:17.910610    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 19:41:18.009235 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:18.009201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" event={"ID":"8f0bde55-a4c4-4831-8cde-0b151cc2d864","Type":"ContainerStarted","Data":"4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2"}
Sep 17 19:41:18.873565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:18.873537    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:41:19.015965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.015935    2583 generic.go:356] "Generic (PLEG): container finished" podID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerID="781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced" exitCode=0
Sep 17 19:41:19.016426 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.015995    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"
Sep 17 19:41:19.016426 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.015998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" event={"ID":"ead24ff9-e971-4443-940b-e1cd9afdbdbf","Type":"ContainerDied","Data":"781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced"}
Sep 17 19:41:19.016426 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.016076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp" event={"ID":"ead24ff9-e971-4443-940b-e1cd9afdbdbf","Type":"ContainerDied","Data":"9029238171eea27760e96b39e897e1b511608c5fcae5b0fac9a58dea12fb46e1"}
Sep 17 19:41:19.016426 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.016121    2583 scope.go:117] "RemoveContainer" containerID="6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd"
Sep 17 19:41:19.024539 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.024524    2583 scope.go:117] "RemoveContainer" containerID="781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced"
Sep 17 19:41:19.031912 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.031897    2583 scope.go:117] "RemoveContainer" containerID="68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec"
Sep 17 19:41:19.039660 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.039638    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a62bdf6-b45b-4a36-940b-d618fbab5e8c" path="/var/lib/kubelet/pods/5a62bdf6-b45b-4a36-940b-d618fbab5e8c/volumes"
Sep 17 19:41:19.039891 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.039870    2583 scope.go:117] "RemoveContainer" containerID="6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd"
Sep 17 19:41:19.040160 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:41:19.040139    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd\": container with ID starting with 6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd not found: ID does not exist" containerID="6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd"
Sep 17 19:41:19.040242 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.040166    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd"} err="failed to get container status \"6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd\": rpc error: code = NotFound desc = could not find container \"6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd\": container with ID starting with 6248c2fdd515ff50b0dc0163dda65ec93b58da9547e95f761cb2fa20d6875dcd not found: ID does not exist"
Sep 17 19:41:19.040242 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.040184    2583 scope.go:117] "RemoveContainer" containerID="781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced"
Sep 17 19:41:19.040454 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:41:19.040435    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced\": container with ID starting with 781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced not found: ID does not exist" containerID="781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced"
Sep 17 19:41:19.040518 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.040467    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced"} err="failed to get container status \"781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced\": rpc error: code = NotFound desc = could not find container \"781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced\": container with ID starting with 781a97065f7556c89b7871a96f329f4fff6c121b8d398c07037f5bcf6bd37ced not found: ID does not exist"
Sep 17 19:41:19.040518 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.040485    2583 scope.go:117] "RemoveContainer" containerID="68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec"
Sep 17 19:41:19.040772 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:41:19.040752    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec\": container with ID starting with 68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec not found: ID does not exist" containerID="68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec"
Sep 17 19:41:19.040849 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.040776    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec"} err="failed to get container status \"68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec\": rpc error: code = NotFound desc = could not find container \"68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec\": container with ID starting with 68ca0465ac352860f1563c3d9d3774237f48d7863fe61e9c39f0fb55cd3759ec not found: ID does not exist"
Sep 17 19:41:19.053953 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.053932    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv72d\" (UniqueName: \"kubernetes.io/projected/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-api-access-sv72d\") pod \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") "
Sep 17 19:41:19.054025 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.053995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kserve-provision-location\") pod \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") "
Sep 17 19:41:19.054073 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.054032    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead24ff9-e971-4443-940b-e1cd9afdbdbf-proxy-tls\") pod \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") "
Sep 17 19:41:19.054073 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.054054    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-rbac-proxy-sar-config\") pod \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") "
Sep 17 19:41:19.054148 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.054081    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-cabundle-cert\") pod \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\" (UID: \"ead24ff9-e971-4443-940b-e1cd9afdbdbf\") "
Sep 17 19:41:19.054447 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.054402    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ead24ff9-e971-4443-940b-e1cd9afdbdbf" (UID: "ead24ff9-e971-4443-940b-e1cd9afdbdbf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:41:19.054559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.054459    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ead24ff9-e971-4443-940b-e1cd9afdbdbf" (UID: "ead24ff9-e971-4443-940b-e1cd9afdbdbf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:19.054658 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.054557    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.054658 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.054578    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.054746 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.054666    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ead24ff9-e971-4443-940b-e1cd9afdbdbf" (UID: "ead24ff9-e971-4443-940b-e1cd9afdbdbf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:41:19.056014 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.055995    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-api-access-sv72d" (OuterVolumeSpecName: "kube-api-access-sv72d") pod "ead24ff9-e971-4443-940b-e1cd9afdbdbf" (UID: "ead24ff9-e971-4443-940b-e1cd9afdbdbf"). InnerVolumeSpecName "kube-api-access-sv72d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:41:19.056219 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.056200    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead24ff9-e971-4443-940b-e1cd9afdbdbf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ead24ff9-e971-4443-940b-e1cd9afdbdbf" (UID: "ead24ff9-e971-4443-940b-e1cd9afdbdbf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:41:19.155268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.155231    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ead24ff9-e971-4443-940b-e1cd9afdbdbf-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.155268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.155260    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead24ff9-e971-4443-940b-e1cd9afdbdbf-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.155268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.155273    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv72d\" (UniqueName: \"kubernetes.io/projected/ead24ff9-e971-4443-940b-e1cd9afdbdbf-kube-api-access-sv72d\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:41:19.339089 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.339054    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"]
Sep 17 19:41:19.345037 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:19.345009    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-stxwp"]
Sep 17 19:41:21.027494 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.027459    2583 generic.go:356] "Generic (PLEG): container finished" podID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerID="4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2" exitCode=0
Sep 17 19:41:21.027991 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.027538    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" event={"ID":"8f0bde55-a4c4-4831-8cde-0b151cc2d864","Type":"ContainerDied","Data":"4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2"}
Sep 17 19:41:21.038550 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.038526    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" path="/var/lib/kubelet/pods/ead24ff9-e971-4443-940b-e1cd9afdbdbf/volumes"
Sep 17 19:41:21.353265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.353191    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"]
Sep 17 19:41:21.353515 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.353504    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kserve-container"
Sep 17 19:41:21.353558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.353516    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kserve-container"
Sep 17 19:41:21.353558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.353530    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="storage-initializer"
Sep 17 19:41:21.353558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.353536    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="storage-initializer"
Sep 17 19:41:21.353558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.353546    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kube-rbac-proxy"
Sep 17 19:41:21.353558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.353551    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kube-rbac-proxy"
Sep 17 19:41:21.353734 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.353604    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kube-rbac-proxy"
Sep 17 19:41:21.353734 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.353636    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead24ff9-e971-4443-940b-e1cd9afdbdbf" containerName="kserve-container"
Sep 17 19:41:21.355457 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.355442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.357561 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.357542    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 19:41:21.357758 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.357743    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 19:41:21.369773 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.369748    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.369903 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.369803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvtnr\" (UniqueName: \"kubernetes.io/projected/21421273-3e90-482b-bfe5-4d51907e7892-kube-api-access-lvtnr\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.369903 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.369857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21421273-3e90-482b-bfe5-4d51907e7892-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.370032 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.369924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.370032 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.369960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21421273-3e90-482b-bfe5-4d51907e7892-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.371504 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.371481    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"]
Sep 17 19:41:21.470482 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.470445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21421273-3e90-482b-bfe5-4d51907e7892-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.470670 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.470498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.470670 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.470526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21421273-3e90-482b-bfe5-4d51907e7892-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.470670 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.470554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.470670 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:41:21.470582    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Sep 17 19:41:21.470670 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.470590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtnr\" (UniqueName: \"kubernetes.io/projected/21421273-3e90-482b-bfe5-4d51907e7892-kube-api-access-lvtnr\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.470670 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:41:21.470666    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21421273-3e90-482b-bfe5-4d51907e7892-proxy-tls podName:21421273-3e90-482b-bfe5-4d51907e7892 nodeName:}" failed. No retries permitted until 2026-09-17 19:41:21.970645408 +0000 UTC m=+2101.098386793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21421273-3e90-482b-bfe5-4d51907e7892-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-r6ljn" (UID: "21421273-3e90-482b-bfe5-4d51907e7892") : secret "isvc-xgboost-predictor-serving-cert" not found
Sep 17 19:41:21.470997 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.470975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21421273-3e90-482b-bfe5-4d51907e7892-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.471219 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.471202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.471256 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.471236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.478922 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.478898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtnr\" (UniqueName: \"kubernetes.io/projected/21421273-3e90-482b-bfe5-4d51907e7892-kube-api-access-lvtnr\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.973267 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.973223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21421273-3e90-482b-bfe5-4d51907e7892-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:21.975745 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:21.975726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21421273-3e90-482b-bfe5-4d51907e7892-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-r6ljn\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:22.034516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:22.034483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" event={"ID":"8f0bde55-a4c4-4831-8cde-0b151cc2d864","Type":"ContainerStarted","Data":"98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889"}
Sep 17 19:41:22.034890 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:22.034549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" event={"ID":"8f0bde55-a4c4-4831-8cde-0b151cc2d864","Type":"ContainerStarted","Data":"3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3"}
Sep 17 19:41:22.034890 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:22.034827    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:22.034997 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:22.034977    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:22.036191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:22.036167    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:41:22.055071 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:22.055022    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podStartSLOduration=2.264044619 podStartE2EDuration="6.055007559s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.79096294s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:41:22.053941831 +0000 UTC m=+2101.181683234" watchObservedRunningTime="2026-09-17 19:41:22.055007559 +0000 UTC m=+2101.182748960"
Sep 17 19:41:22.265000 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:22.264916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:22.535202 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:22.535177    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"]
Sep 17 19:41:22.537794 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:41:22.537767    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21421273_3e90_482b_bfe5_4d51907e7892.slice/crio-ef6a7027db6365011666494b1a1a1b37787990848f50e97f7909db173f8c1a8e WatchSource:0}: Error finding container ef6a7027db6365011666494b1a1a1b37787990848f50e97f7909db173f8c1a8e: Status 404 returned error can't find the container with id ef6a7027db6365011666494b1a1a1b37787990848f50e97f7909db173f8c1a8e
Sep 17 19:41:23.041520 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:23.041488    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" event={"ID":"21421273-3e90-482b-bfe5-4d51907e7892","Type":"ContainerStarted","Data":"d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3"}
Sep 17 19:41:23.041978 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:23.041544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" event={"ID":"21421273-3e90-482b-bfe5-4d51907e7892","Type":"ContainerStarted","Data":"ef6a7027db6365011666494b1a1a1b37787990848f50e97f7909db173f8c1a8e"}
Sep 17 19:41:23.041978 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:23.041869    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:41:27.058456 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:27.058422    2583 generic.go:356] "Generic (PLEG): container finished" podID="21421273-3e90-482b-bfe5-4d51907e7892" containerID="d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3" exitCode=0
Sep 17 19:41:27.058834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:27.058494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" event={"ID":"21421273-3e90-482b-bfe5-4d51907e7892","Type":"ContainerDied","Data":"d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3"}
Sep 17 19:41:27.821849 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:27.821816    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:41:27.823689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:27.823656    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:41:28.046258 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:28.046230    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:41:28.046835 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:28.046802    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:41:38.047335 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:38.047287    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:41:48.046847 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:48.046802    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:41:48.164756 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:48.164722    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" event={"ID":"21421273-3e90-482b-bfe5-4d51907e7892","Type":"ContainerStarted","Data":"d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f"}
Sep 17 19:41:48.164907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:48.164783    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" event={"ID":"21421273-3e90-482b-bfe5-4d51907e7892","Type":"ContainerStarted","Data":"c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652"}
Sep 17 19:41:48.165071 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:48.165044    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:48.165206 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:48.165176    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:48.166317 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:48.166293    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 19:41:48.184029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:48.183991    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podStartSLOduration=3.241986599 podStartE2EDuration="27.183979768s" totalImagesPullingTime="20.129479859s" totalInitContainerRuntime="3.81251331s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:41:48.182685778 +0000 UTC m=+2127.310427180" watchObservedRunningTime="2026-09-17 19:41:48.183979768 +0000 UTC m=+2127.311721168"
Sep 17 19:41:49.169007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:49.168969    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 19:41:54.173331 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:54.173301    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:41:54.173882 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:54.173857    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 19:41:58.046887 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:41:58.046846    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:42:04.174396 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:04.174297    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 19:42:08.047322 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:08.047276    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:42:14.174107 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:14.174068    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 19:42:18.047283 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:18.047246    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:42:24.173792 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:24.173751    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 19:42:28.047594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:28.047548    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:42:34.173850 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:34.173800    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 19:42:38.047803 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:38.047756    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:42:39.189370 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:39.189336    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"]
Sep 17 19:42:39.189817 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:39.189565    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" containerID="cri-o://3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3" gracePeriod=30
Sep 17 19:42:39.189817 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:39.189654    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kube-rbac-proxy" containerID="cri-o://98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889" gracePeriod=30
Sep 17 19:42:39.189935 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:39.189821    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 19:42:39.389269 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:39.389226    2583 generic.go:356] "Generic (PLEG): container finished" podID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerID="98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889" exitCode=2
Sep 17 19:42:39.389449 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:39.389285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" event={"ID":"8f0bde55-a4c4-4831-8cde-0b151cc2d864","Type":"ContainerDied","Data":"98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889"}
Sep 17 19:42:43.042025 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:43.041987    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 19:42:44.174650 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.174598    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 19:42:44.832868 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.832846    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:42:44.994818 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.994781    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-cabundle-cert\") pod \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") "
Sep 17 19:42:44.995010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.994832    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f0bde55-a4c4-4831-8cde-0b151cc2d864-proxy-tls\") pod \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") "
Sep 17 19:42:44.995010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.994857    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kserve-provision-location\") pod \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") "
Sep 17 19:42:44.995010 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.994926    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-rbac-proxy-sar-config\") pod \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") "
Sep 17 19:42:44.995189 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.995036    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcgvf\" (UniqueName: \"kubernetes.io/projected/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-api-access-gcgvf\") pod \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\" (UID: \"8f0bde55-a4c4-4831-8cde-0b151cc2d864\") "
Sep 17 19:42:44.995302 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.995229    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f0bde55-a4c4-4831-8cde-0b151cc2d864" (UID: "8f0bde55-a4c4-4831-8cde-0b151cc2d864"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:42:44.995302 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.995237    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f0bde55-a4c4-4831-8cde-0b151cc2d864" (UID: "8f0bde55-a4c4-4831-8cde-0b151cc2d864"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:42:44.995302 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.995261    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f0bde55-a4c4-4831-8cde-0b151cc2d864" (UID: "8f0bde55-a4c4-4831-8cde-0b151cc2d864"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:42:44.995427 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.995358    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:42:44.995427 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.995370    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f0bde55-a4c4-4831-8cde-0b151cc2d864-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:42:44.995427 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.995384    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:42:44.997237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.997213    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0bde55-a4c4-4831-8cde-0b151cc2d864-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f0bde55-a4c4-4831-8cde-0b151cc2d864" (UID: "8f0bde55-a4c4-4831-8cde-0b151cc2d864"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:42:44.997237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:44.997226    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-api-access-gcgvf" (OuterVolumeSpecName: "kube-api-access-gcgvf") pod "8f0bde55-a4c4-4831-8cde-0b151cc2d864" (UID: "8f0bde55-a4c4-4831-8cde-0b151cc2d864"). InnerVolumeSpecName "kube-api-access-gcgvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:42:45.096611 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.096533    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcgvf\" (UniqueName: \"kubernetes.io/projected/8f0bde55-a4c4-4831-8cde-0b151cc2d864-kube-api-access-gcgvf\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:42:45.096611 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.096563    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f0bde55-a4c4-4831-8cde-0b151cc2d864-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:42:45.417450 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.417419    2583 generic.go:356] "Generic (PLEG): container finished" podID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerID="3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3" exitCode=139
Sep 17 19:42:45.417941 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.417491    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"
Sep 17 19:42:45.417941 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.417497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" event={"ID":"8f0bde55-a4c4-4831-8cde-0b151cc2d864","Type":"ContainerDied","Data":"3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3"}
Sep 17 19:42:45.417941 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.417557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq" event={"ID":"8f0bde55-a4c4-4831-8cde-0b151cc2d864","Type":"ContainerDied","Data":"ac99229eb8abf7978f2b865389921a3c3ff332d7ea62443380f8ecbf5999a5ff"}
Sep 17 19:42:45.417941 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.417640    2583 scope.go:117] "RemoveContainer" containerID="98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889"
Sep 17 19:42:45.425926 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.425907    2583 scope.go:117] "RemoveContainer" containerID="3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3"
Sep 17 19:42:45.433400 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.433374    2583 scope.go:117] "RemoveContainer" containerID="4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2"
Sep 17 19:42:45.438852 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.438829    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"]
Sep 17 19:42:45.443375 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.443355    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-4h2dq"]
Sep 17 19:42:45.444259 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.444243    2583 scope.go:117] "RemoveContainer" containerID="98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889"
Sep 17 19:42:45.444551 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:42:45.444533    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889\": container with ID starting with 98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889 not found: ID does not exist" containerID="98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889"
Sep 17 19:42:45.444717 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.444561    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889"} err="failed to get container status \"98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889\": rpc error: code = NotFound desc = could not find container \"98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889\": container with ID starting with 98dfccb6f8ebab621bd82cb747dcc624146e6b228126774e169bfb6640187889 not found: ID does not exist"
Sep 17 19:42:45.444717 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.444585    2583 scope.go:117] "RemoveContainer" containerID="3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3"
Sep 17 19:42:45.444891 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:42:45.444875    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3\": container with ID starting with 3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3 not found: ID does not exist" containerID="3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3"
Sep 17 19:42:45.444936 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.444896    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3"} err="failed to get container status \"3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3\": rpc error: code = NotFound desc = could not find container \"3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3\": container with ID starting with 3ebc0da54bdf92f2e45e1bb822cba7b8c2e1005f41aa17d9a54c39b5a34805d3 not found: ID does not exist"
Sep 17 19:42:45.444936 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.444910    2583 scope.go:117] "RemoveContainer" containerID="4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2"
Sep 17 19:42:45.445160 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:42:45.445140    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2\": container with ID starting with 4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2 not found: ID does not exist" containerID="4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2"
Sep 17 19:42:45.445238 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:45.445162    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2"} err="failed to get container status \"4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2\": rpc error: code = NotFound desc = could not find container \"4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2\": container with ID starting with 4b12addc18bed6a47b988e4c94d60aded612cee9cda56397771ffff786ccfdd2 not found: ID does not exist"
Sep 17 19:42:47.038328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:47.038289    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" path="/var/lib/kubelet/pods/8f0bde55-a4c4-4831-8cde-0b151cc2d864/volumes"
Sep 17 19:42:54.175206 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:54.175178    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:42:56.580793 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.580759    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"]
Sep 17 19:42:56.581141 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.581071    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="storage-initializer"
Sep 17 19:42:56.581141 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.581081    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="storage-initializer"
Sep 17 19:42:56.581141 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.581094    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kube-rbac-proxy"
Sep 17 19:42:56.581141 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.581100    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kube-rbac-proxy"
Sep 17 19:42:56.581141 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.581112    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container"
Sep 17 19:42:56.581141 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.581117    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container"
Sep 17 19:42:56.581383 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.581192    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kube-rbac-proxy"
Sep 17 19:42:56.581383 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.581200    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f0bde55-a4c4-4831-8cde-0b151cc2d864" containerName="kserve-container"
Sep 17 19:42:56.583339 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.583323    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.585769 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.585747    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 19:42:56.585880 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.585748    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 19:42:56.589062 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.589035    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89xl\" (UniqueName: \"kubernetes.io/projected/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-api-access-v89xl\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.589175 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.589083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.589175 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.589117    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.589175 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.589151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ec5d8d-023b-43b2-934e-df7706831ace-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.589324 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.589177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ec5d8d-023b-43b2-934e-df7706831ace-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.593998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.593974    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"]
Sep 17 19:42:56.689884 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.689849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89xl\" (UniqueName: \"kubernetes.io/projected/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-api-access-v89xl\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.690057 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.689891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.690057 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.689925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.690057 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.689959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ec5d8d-023b-43b2-934e-df7706831ace-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.690057 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.689985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ec5d8d-023b-43b2-934e-df7706831ace-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.690360 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.690339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ec5d8d-023b-43b2-934e-df7706831ace-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.690629 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.690591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.690688 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.690643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.692572 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.692543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ec5d8d-023b-43b2-934e-df7706831ace-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.698488 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.698459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89xl\" (UniqueName: \"kubernetes.io/projected/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-api-access-v89xl\") pod \"isvc-lightgbm-predictor-54f85ddf88-8jmrz\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:56.894204 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:56.894118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:42:57.167157 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:57.167064    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"]
Sep 17 19:42:57.169419 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:42:57.169395    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ec5d8d_023b_43b2_934e_df7706831ace.slice/crio-e663d2ec99e4ee561fe9f33b807ee1aa498acf3a82edb65db3d68bcde3f8e9cb WatchSource:0}: Error finding container e663d2ec99e4ee561fe9f33b807ee1aa498acf3a82edb65db3d68bcde3f8e9cb: Status 404 returned error can't find the container with id e663d2ec99e4ee561fe9f33b807ee1aa498acf3a82edb65db3d68bcde3f8e9cb
Sep 17 19:42:57.470668 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:57.470635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" event={"ID":"d4ec5d8d-023b-43b2-934e-df7706831ace","Type":"ContainerStarted","Data":"c4fd1a4f23924c7ee4cfa3263951236461d986ae81d6d0edade93553d8e972ec"}
Sep 17 19:42:57.470830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:42:57.470698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" event={"ID":"d4ec5d8d-023b-43b2-934e-df7706831ace","Type":"ContainerStarted","Data":"e663d2ec99e4ee561fe9f33b807ee1aa498acf3a82edb65db3d68bcde3f8e9cb"}
Sep 17 19:43:01.489491 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:01.489455    2583 generic.go:356] "Generic (PLEG): container finished" podID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerID="c4fd1a4f23924c7ee4cfa3263951236461d986ae81d6d0edade93553d8e972ec" exitCode=0
Sep 17 19:43:01.489491 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:01.489495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" event={"ID":"d4ec5d8d-023b-43b2-934e-df7706831ace","Type":"ContainerDied","Data":"c4fd1a4f23924c7ee4cfa3263951236461d986ae81d6d0edade93553d8e972ec"}
Sep 17 19:43:01.490710 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:01.490691    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:43:01.869159 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:01.869066    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"]
Sep 17 19:43:01.869471 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:01.869382    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" containerID="cri-o://c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652" gracePeriod=30
Sep 17 19:43:01.869642 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:01.869489    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kube-rbac-proxy" containerID="cri-o://d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f" gracePeriod=30
Sep 17 19:43:02.496120 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:02.496086    2583 generic.go:356] "Generic (PLEG): container finished" podID="21421273-3e90-482b-bfe5-4d51907e7892" containerID="d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f" exitCode=2
Sep 17 19:43:02.496493 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:02.496157    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" event={"ID":"21421273-3e90-482b-bfe5-4d51907e7892","Type":"ContainerDied","Data":"d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f"}
Sep 17 19:43:04.169252 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:04.169206    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 17 19:43:04.173993 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:04.173956    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 19:43:07.060903 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.060262    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"]
Sep 17 19:43:07.077798 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.076997    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"]
Sep 17 19:43:07.077798 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.077138    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.079751 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.079723    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 19:43:07.080563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.080239    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 19:43:07.185401 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.185358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.185567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.185412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.185567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.185500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.185715 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.185575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.185715 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.185601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56gc\" (UniqueName: \"kubernetes.io/projected/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-api-access-p56gc\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.286332 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.286291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.286524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.286348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.286524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.286392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.286524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.286447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.286524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.286471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56gc\" (UniqueName: \"kubernetes.io/projected/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-api-access-p56gc\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.286783 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:43:07.286601    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 19:43:07.286783 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:43:07.286712    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-proxy-tls podName:d9ca4a5a-0b68-42cb-80f7-02b98bc2008e nodeName:}" failed. No retries permitted until 2026-09-17 19:43:07.786688911 +0000 UTC m=+2206.914430296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" (UID: "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 19:43:07.286896 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.286821    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.287253 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.287225    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.287368 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.287354    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.296221 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.296193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56gc\" (UniqueName: \"kubernetes.io/projected/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-api-access-p56gc\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.791448 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.791408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.794006 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.793972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:07.993835 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:07.993800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:08.025068 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.025045    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:43:08.093459 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.093437    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21421273-3e90-482b-bfe5-4d51907e7892-kserve-provision-location\") pod \"21421273-3e90-482b-bfe5-4d51907e7892\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") "
Sep 17 19:43:08.093848 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.093501    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-cabundle-cert\") pod \"21421273-3e90-482b-bfe5-4d51907e7892\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") "
Sep 17 19:43:08.093848 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.093534    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvtnr\" (UniqueName: \"kubernetes.io/projected/21421273-3e90-482b-bfe5-4d51907e7892-kube-api-access-lvtnr\") pod \"21421273-3e90-482b-bfe5-4d51907e7892\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") "
Sep 17 19:43:08.093848 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.093554    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21421273-3e90-482b-bfe5-4d51907e7892-proxy-tls\") pod \"21421273-3e90-482b-bfe5-4d51907e7892\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") "
Sep 17 19:43:08.093848 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.093629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-kube-rbac-proxy-sar-config\") pod \"21421273-3e90-482b-bfe5-4d51907e7892\" (UID: \"21421273-3e90-482b-bfe5-4d51907e7892\") "
Sep 17 19:43:08.093848 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.093737    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21421273-3e90-482b-bfe5-4d51907e7892-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21421273-3e90-482b-bfe5-4d51907e7892" (UID: "21421273-3e90-482b-bfe5-4d51907e7892"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:08.094135 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.093890    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21421273-3e90-482b-bfe5-4d51907e7892-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:43:08.094135 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.094021    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21421273-3e90-482b-bfe5-4d51907e7892" (UID: "21421273-3e90-482b-bfe5-4d51907e7892"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:43:08.094135 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.094065    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21421273-3e90-482b-bfe5-4d51907e7892" (UID: "21421273-3e90-482b-bfe5-4d51907e7892"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:43:08.095950 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.095927    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21421273-3e90-482b-bfe5-4d51907e7892-kube-api-access-lvtnr" (OuterVolumeSpecName: "kube-api-access-lvtnr") pod "21421273-3e90-482b-bfe5-4d51907e7892" (UID: "21421273-3e90-482b-bfe5-4d51907e7892"). InnerVolumeSpecName "kube-api-access-lvtnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:43:08.096042 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.096002    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21421273-3e90-482b-bfe5-4d51907e7892-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21421273-3e90-482b-bfe5-4d51907e7892" (UID: "21421273-3e90-482b-bfe5-4d51907e7892"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:43:08.195325 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.195284    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:43:08.195325 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.195316    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21421273-3e90-482b-bfe5-4d51907e7892-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:43:08.195325 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.195329    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvtnr\" (UniqueName: \"kubernetes.io/projected/21421273-3e90-482b-bfe5-4d51907e7892-kube-api-access-lvtnr\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:43:08.195561 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.195341    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21421273-3e90-482b-bfe5-4d51907e7892-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:43:08.283069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.283043    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"]
Sep 17 19:43:08.284975 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:43:08.284946    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ca4a5a_0b68_42cb_80f7_02b98bc2008e.slice/crio-e7a808088655a5965dca68986dd114b47f398c1554ba47bf640fa468b4b57697 WatchSource:0}: Error finding container e7a808088655a5965dca68986dd114b47f398c1554ba47bf640fa468b4b57697: Status 404 returned error can't find the container with id e7a808088655a5965dca68986dd114b47f398c1554ba47bf640fa468b4b57697
Sep 17 19:43:08.529047 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.529014    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" event={"ID":"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e","Type":"ContainerStarted","Data":"1ce53d2cc33bebc8eb919a98c03d2cc9751c45353ed5b371ee74ce77c627fd26"}
Sep 17 19:43:08.529270 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.529095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" event={"ID":"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e","Type":"ContainerStarted","Data":"e7a808088655a5965dca68986dd114b47f398c1554ba47bf640fa468b4b57697"}
Sep 17 19:43:08.531398 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.531369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" event={"ID":"d4ec5d8d-023b-43b2-934e-df7706831ace","Type":"ContainerStarted","Data":"7eecddb0df60f41c3580ec95cd805a558e931783a0d6f532a523d38a485a2056"}
Sep 17 19:43:08.531502 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.531434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" event={"ID":"d4ec5d8d-023b-43b2-934e-df7706831ace","Type":"ContainerStarted","Data":"97ba449df4da7ecd46ceb6c27582600f84d8794fa79c96ad6fefbd95a947f6f3"}
Sep 17 19:43:08.531707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.531682    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:43:08.531802 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.531734    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:43:08.533485 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.533456    2583 generic.go:356] "Generic (PLEG): container finished" podID="21421273-3e90-482b-bfe5-4d51907e7892" containerID="c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652" exitCode=0
Sep 17 19:43:08.533579 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.533506    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" event={"ID":"21421273-3e90-482b-bfe5-4d51907e7892","Type":"ContainerDied","Data":"c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652"}
Sep 17 19:43:08.533579 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.533509    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:43:08.533579 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.533555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn" event={"ID":"21421273-3e90-482b-bfe5-4d51907e7892","Type":"ContainerDied","Data":"ef6a7027db6365011666494b1a1a1b37787990848f50e97f7909db173f8c1a8e"}
Sep 17 19:43:08.533579 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.533511    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"
Sep 17 19:43:08.533755 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.533594    2583 scope.go:117] "RemoveContainer" containerID="d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f"
Sep 17 19:43:08.543003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.542985    2583 scope.go:117] "RemoveContainer" containerID="c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652"
Sep 17 19:43:08.550923 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.550903    2583 scope.go:117] "RemoveContainer" containerID="d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3"
Sep 17 19:43:08.558351 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.558335    2583 scope.go:117] "RemoveContainer" containerID="d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f"
Sep 17 19:43:08.558662 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:43:08.558642    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f\": container with ID starting with d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f not found: ID does not exist" containerID="d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f"
Sep 17 19:43:08.558749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.558674    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f"} err="failed to get container status \"d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f\": rpc error: code = NotFound desc = could not find container \"d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f\": container with ID starting with d5c821fe76485e5fd28f01c89e171d51fae5f8bbdec59dc9204079ceef0c9c7f not found: ID does not exist"
Sep 17 19:43:08.558749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.558697    2583 scope.go:117] "RemoveContainer" containerID="c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652"
Sep 17 19:43:08.558958 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:43:08.558943    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652\": container with ID starting with c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652 not found: ID does not exist" containerID="c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652"
Sep 17 19:43:08.559021 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.558967    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652"} err="failed to get container status \"c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652\": rpc error: code = NotFound desc = could not find container \"c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652\": container with ID starting with c39c81bf5d8a4e7622fa656f3bfcf9886d49d81d837a5f3c271bcc9168636652 not found: ID does not exist"
Sep 17 19:43:08.559021 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.558986    2583 scope.go:117] "RemoveContainer" containerID="d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3"
Sep 17 19:43:08.559240 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:43:08.559226    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3\": container with ID starting with d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3 not found: ID does not exist" containerID="d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3"
Sep 17 19:43:08.559293 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.559246    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3"} err="failed to get container status \"d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3\": rpc error: code = NotFound desc = could not find container \"d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3\": container with ID starting with d907d92fd9d8d9c27c019b00cd28471fc0441ad8459069e8a3e3bedd914129b3 not found: ID does not exist"
Sep 17 19:43:08.572787 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.572748    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podStartSLOduration=2.225695002 podStartE2EDuration="12.572734834s" totalImagesPullingTime="6.53655179s" totalInitContainerRuntime="3.810488042s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:43:08.570917136 +0000 UTC m=+2207.698658550" watchObservedRunningTime="2026-09-17 19:43:08.572734834 +0000 UTC m=+2207.700476234"
Sep 17 19:43:08.585763 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.585742    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"]
Sep 17 19:43:08.591128 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:08.591103    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6ljn"]
Sep 17 19:43:09.039478 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:09.039433    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21421273-3e90-482b-bfe5-4d51907e7892" path="/var/lib/kubelet/pods/21421273-3e90-482b-bfe5-4d51907e7892/volumes"
Sep 17 19:43:09.539473 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:09.539435    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:43:12.558714 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:12.558602    2583 generic.go:356] "Generic (PLEG): container finished" podID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerID="1ce53d2cc33bebc8eb919a98c03d2cc9751c45353ed5b371ee74ce77c627fd26" exitCode=0
Sep 17 19:43:12.558714 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:12.558651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" event={"ID":"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e","Type":"ContainerDied","Data":"1ce53d2cc33bebc8eb919a98c03d2cc9751c45353ed5b371ee74ce77c627fd26"}
Sep 17 19:43:13.565833 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:13.565801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" event={"ID":"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e","Type":"ContainerStarted","Data":"bc2738e04a49c443ed7f79039c3d20585637f5b3ed758da1c44c2b399dbad1c9"}
Sep 17 19:43:13.566190 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:13.565867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" event={"ID":"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e","Type":"ContainerStarted","Data":"bd99156e3cf3b74c33888119ff4d675b361b49838cd8bdbc57f994bba6de715f"}
Sep 17 19:43:13.566190 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:13.566064    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:13.595150 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:13.595108    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" podStartSLOduration=2.723095909 podStartE2EDuration="6.595095168s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.871999259s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:43:13.594785956 +0000 UTC m=+2212.722527359" watchObservedRunningTime="2026-09-17 19:43:13.595095168 +0000 UTC m=+2212.722836568"
Sep 17 19:43:14.543867 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:14.543835    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:43:14.544359 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:14.544326    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:43:14.571757 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:14.571731    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:20.580326 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:20.580296    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:24.544731 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:24.544684    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:43:34.545274 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:34.545183    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:43:44.544527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:44.544483    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:43:50.585025 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:50.584992    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:43:54.544450 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:54.544402    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:43:57.121080 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:57.121049    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"]
Sep 17 19:43:57.121455 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:57.121296    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kserve-container" containerID="cri-o://bd99156e3cf3b74c33888119ff4d675b361b49838cd8bdbc57f994bba6de715f" gracePeriod=30
Sep 17 19:43:57.121455 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:57.121326    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kube-rbac-proxy" containerID="cri-o://bc2738e04a49c443ed7f79039c3d20585637f5b3ed758da1c44c2b399dbad1c9" gracePeriod=30
Sep 17 19:43:57.763105 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:57.763073    2583 generic.go:356] "Generic (PLEG): container finished" podID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerID="bc2738e04a49c443ed7f79039c3d20585637f5b3ed758da1c44c2b399dbad1c9" exitCode=2
Sep 17 19:43:57.763276 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:43:57.763141    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" event={"ID":"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e","Type":"ContainerDied","Data":"bc2738e04a49c443ed7f79039c3d20585637f5b3ed758da1c44c2b399dbad1c9"}
Sep 17 19:44:00.576022 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:00.575980    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 17 19:44:00.581410 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:00.581380    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.57:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 19:44:02.306196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.306162    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"]
Sep 17 19:44:02.306559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.306507    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="storage-initializer"
Sep 17 19:44:02.306559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.306518    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="storage-initializer"
Sep 17 19:44:02.306559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.306527    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container"
Sep 17 19:44:02.306559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.306533    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container"
Sep 17 19:44:02.306559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.306548    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kube-rbac-proxy"
Sep 17 19:44:02.306559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.306553    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kube-rbac-proxy"
Sep 17 19:44:02.306814 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.306605    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kserve-container"
Sep 17 19:44:02.306814 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.306629    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="21421273-3e90-482b-bfe5-4d51907e7892" containerName="kube-rbac-proxy"
Sep 17 19:44:02.310021 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.310002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.312500 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.312472    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 19:44:02.312606 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.312475    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 19:44:02.321066 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.321044    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"]
Sep 17 19:44:02.448521 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.448476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ggz\" (UniqueName: \"kubernetes.io/projected/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-api-access-w4ggz\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.448749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.448532    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.448749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.448565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff8533a-bead-44d0-831d-205e5b5e888f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.448749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.448583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.448749 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.448646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff8533a-bead-44d0-831d-205e5b5e888f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.550083 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.550051    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff8533a-bead-44d0-831d-205e5b5e888f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.550287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.550104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ggz\" (UniqueName: \"kubernetes.io/projected/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-api-access-w4ggz\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.550287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.550145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.550287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.550171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff8533a-bead-44d0-831d-205e5b5e888f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.550287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.550196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.550568 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.550547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff8533a-bead-44d0-831d-205e5b5e888f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.550987 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.550965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.551126 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.551104    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.552903 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.552885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff8533a-bead-44d0-831d-205e5b5e888f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.558226 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.558171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ggz\" (UniqueName: \"kubernetes.io/projected/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-api-access-w4ggz\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.620046 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.620019    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:02.898072 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:02.897997    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"]
Sep 17 19:44:02.901258 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:44:02.901233    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff8533a_bead_44d0_831d_205e5b5e888f.slice/crio-7151f3ad9a49291788112e34a817aba78fc7ef6c76d84a57a1128214bddd6f7a WatchSource:0}: Error finding container 7151f3ad9a49291788112e34a817aba78fc7ef6c76d84a57a1128214bddd6f7a: Status 404 returned error can't find the container with id 7151f3ad9a49291788112e34a817aba78fc7ef6c76d84a57a1128214bddd6f7a
Sep 17 19:44:03.794149 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.794114    2583 generic.go:356] "Generic (PLEG): container finished" podID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerID="bd99156e3cf3b74c33888119ff4d675b361b49838cd8bdbc57f994bba6de715f" exitCode=0
Sep 17 19:44:03.794529 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.794143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" event={"ID":"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e","Type":"ContainerDied","Data":"bd99156e3cf3b74c33888119ff4d675b361b49838cd8bdbc57f994bba6de715f"}
Sep 17 19:44:03.795957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.795934    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" event={"ID":"2ff8533a-bead-44d0-831d-205e5b5e888f","Type":"ContainerStarted","Data":"4c6077a157ac4c537201490020635db519fcded9f766b16e7edc3c5190bfb100"}
Sep 17 19:44:03.796076 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.795995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" event={"ID":"2ff8533a-bead-44d0-831d-205e5b5e888f","Type":"ContainerStarted","Data":"7151f3ad9a49291788112e34a817aba78fc7ef6c76d84a57a1128214bddd6f7a"}
Sep 17 19:44:03.868939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.868907    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:44:03.961716 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.961685    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-cabundle-cert\") pod \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") "
Sep 17 19:44:03.961918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.961725    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p56gc\" (UniqueName: \"kubernetes.io/projected/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-api-access-p56gc\") pod \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") "
Sep 17 19:44:03.961918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.961757    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-proxy-tls\") pod \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") "
Sep 17 19:44:03.961918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.961785    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-rbac-proxy-sar-config\") pod \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") "
Sep 17 19:44:03.961918 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.961843    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kserve-provision-location\") pod \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\" (UID: \"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e\") "
Sep 17 19:44:03.962134 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.962056    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" (UID: "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:44:03.962189 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.962165    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:03.962189 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.962157    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" (UID: "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:44:03.962268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.962205    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" (UID: "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:03.964031 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.964010    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" (UID: "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:44:03.964108 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:03.964085    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-api-access-p56gc" (OuterVolumeSpecName: "kube-api-access-p56gc") pod "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" (UID: "d9ca4a5a-0b68-42cb-80f7-02b98bc2008e"). InnerVolumeSpecName "kube-api-access-p56gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:44:04.063227 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.063201    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p56gc\" (UniqueName: \"kubernetes.io/projected/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-api-access-p56gc\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:04.063227 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.063226    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:04.063394 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.063236    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:04.063394 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.063245    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:04.544940 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.544895    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:44:04.802009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.801936    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"
Sep 17 19:44:04.802009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.801959    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75" event={"ID":"d9ca4a5a-0b68-42cb-80f7-02b98bc2008e","Type":"ContainerDied","Data":"e7a808088655a5965dca68986dd114b47f398c1554ba47bf640fa468b4b57697"}
Sep 17 19:44:04.802467 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.802040    2583 scope.go:117] "RemoveContainer" containerID="bc2738e04a49c443ed7f79039c3d20585637f5b3ed758da1c44c2b399dbad1c9"
Sep 17 19:44:04.811399 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.811382    2583 scope.go:117] "RemoveContainer" containerID="bd99156e3cf3b74c33888119ff4d675b361b49838cd8bdbc57f994bba6de715f"
Sep 17 19:44:04.819305 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.819288    2583 scope.go:117] "RemoveContainer" containerID="1ce53d2cc33bebc8eb919a98c03d2cc9751c45353ed5b371ee74ce77c627fd26"
Sep 17 19:44:04.823091 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.823071    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"]
Sep 17 19:44:04.827067 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:04.827047    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6vc75"]
Sep 17 19:44:05.038801 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:05.038771    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" path="/var/lib/kubelet/pods/d9ca4a5a-0b68-42cb-80f7-02b98bc2008e/volumes"
Sep 17 19:44:07.818158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:07.818124    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerID="4c6077a157ac4c537201490020635db519fcded9f766b16e7edc3c5190bfb100" exitCode=0
Sep 17 19:44:07.818531 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:07.818171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" event={"ID":"2ff8533a-bead-44d0-831d-205e5b5e888f","Type":"ContainerDied","Data":"4c6077a157ac4c537201490020635db519fcded9f766b16e7edc3c5190bfb100"}
Sep 17 19:44:08.824647 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:08.824600    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" event={"ID":"2ff8533a-bead-44d0-831d-205e5b5e888f","Type":"ContainerStarted","Data":"564125b5d96ce6c80b36860df2db2544310b01b427a236616c2dabed9b075bd8"}
Sep 17 19:44:08.825120 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:08.824687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" event={"ID":"2ff8533a-bead-44d0-831d-205e5b5e888f","Type":"ContainerStarted","Data":"3ac3d10da1269c32e45f064901c465ab81f1f855572fec99992699ce8d67fd56"}
Sep 17 19:44:08.825120 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:08.824904    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:08.844167 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:08.844121    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" podStartSLOduration=2.920186034 podStartE2EDuration="6.844107109s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.923921075s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:44:08.842906155 +0000 UTC m=+2267.970647556" watchObservedRunningTime="2026-09-17 19:44:08.844107109 +0000 UTC m=+2267.971848510"
Sep 17 19:44:09.829604 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:09.829565    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:14.544919 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:14.544869    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:44:15.838950 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:15.838917    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:24.544444 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:24.544396    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:44:34.545185 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:34.545135    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 19:44:44.034819 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:44.034774    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:44:45.843079 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:45.843049    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:47.260004 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:47.259968    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"]
Sep 17 19:44:47.260475 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:47.260298    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container" containerID="cri-o://97ba449df4da7ecd46ceb6c27582600f84d8794fa79c96ad6fefbd95a947f6f3" gracePeriod=30
Sep 17 19:44:47.260475 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:47.260355    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kube-rbac-proxy" containerID="cri-o://7eecddb0df60f41c3580ec95cd805a558e931783a0d6f532a523d38a485a2056" gracePeriod=30
Sep 17 19:44:47.998827 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:47.998794    2583 generic.go:356] "Generic (PLEG): container finished" podID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerID="7eecddb0df60f41c3580ec95cd805a558e931783a0d6f532a523d38a485a2056" exitCode=2
Sep 17 19:44:47.998998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:47.998864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" event={"ID":"d4ec5d8d-023b-43b2-934e-df7706831ace","Type":"ContainerDied","Data":"7eecddb0df60f41c3580ec95cd805a558e931783a0d6f532a523d38a485a2056"}
Sep 17 19:44:49.540110 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:49.540064    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 17 19:44:52.406544 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.406508    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"]
Sep 17 19:44:52.406977 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.406839    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kserve-container" containerID="cri-o://3ac3d10da1269c32e45f064901c465ab81f1f855572fec99992699ce8d67fd56" gracePeriod=30
Sep 17 19:44:52.406977 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.406859    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kube-rbac-proxy" containerID="cri-o://564125b5d96ce6c80b36860df2db2544310b01b427a236616c2dabed9b075bd8" gracePeriod=30
Sep 17 19:44:52.434852 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.434825    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"]
Sep 17 19:44:52.435160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.435149    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kserve-container"
Sep 17 19:44:52.435203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.435162    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kserve-container"
Sep 17 19:44:52.435203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.435171    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kube-rbac-proxy"
Sep 17 19:44:52.435203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.435176    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kube-rbac-proxy"
Sep 17 19:44:52.435203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.435187    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="storage-initializer"
Sep 17 19:44:52.435203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.435193    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="storage-initializer"
Sep 17 19:44:52.435362 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.435253    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kube-rbac-proxy"
Sep 17 19:44:52.435362 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.435263    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ca4a5a-0b68-42cb-80f7-02b98bc2008e" containerName="kserve-container"
Sep 17 19:44:52.442978 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.442956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.445265 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.445232    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 19:44:52.445378 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.445284    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:44:52.449390 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.449362    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"]
Sep 17 19:44:52.584092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.584062    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvgv5\" (UniqueName: \"kubernetes.io/projected/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-api-access-bvgv5\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.584268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.584101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.584268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.584166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.584268 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.584238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.584412 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.584271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.685369 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.685337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.685531 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.685376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.685531 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.685413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvgv5\" (UniqueName: \"kubernetes.io/projected/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-api-access-bvgv5\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.685531 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.685430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.685531 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.685465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.685818 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.685784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.686164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.686143    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.686164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.686154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.688040 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.688020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.695887 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.695867    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvgv5\" (UniqueName: \"kubernetes.io/projected/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-api-access-bvgv5\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:52.754468 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:52.754439    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:53.022008 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.021977    2583 generic.go:356] "Generic (PLEG): container finished" podID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerID="97ba449df4da7ecd46ceb6c27582600f84d8794fa79c96ad6fefbd95a947f6f3" exitCode=0
Sep 17 19:44:53.022178 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.022061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" event={"ID":"d4ec5d8d-023b-43b2-934e-df7706831ace","Type":"ContainerDied","Data":"97ba449df4da7ecd46ceb6c27582600f84d8794fa79c96ad6fefbd95a947f6f3"}
Sep 17 19:44:53.024769 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.024744    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerID="564125b5d96ce6c80b36860df2db2544310b01b427a236616c2dabed9b075bd8" exitCode=2
Sep 17 19:44:53.024924 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.024810    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" event={"ID":"2ff8533a-bead-44d0-831d-205e5b5e888f","Type":"ContainerDied","Data":"564125b5d96ce6c80b36860df2db2544310b01b427a236616c2dabed9b075bd8"}
Sep 17 19:44:53.026093 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.026068    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"]
Sep 17 19:44:53.029841 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:44:53.029818    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9b9f9f_bc90_4ff8_87a5_8e3ce3ba0528.slice/crio-70cbf6a0f0c9449b4ff826a256f51ea08d95568db8d3ae93fdb6e971ba5fc816 WatchSource:0}: Error finding container 70cbf6a0f0c9449b4ff826a256f51ea08d95568db8d3ae93fdb6e971ba5fc816: Status 404 returned error can't find the container with id 70cbf6a0f0c9449b4ff826a256f51ea08d95568db8d3ae93fdb6e971ba5fc816
Sep 17 19:44:53.093053 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.093029    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:44:53.189459 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189422    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-cabundle-cert\") pod \"d4ec5d8d-023b-43b2-934e-df7706831ace\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") "
Sep 17 19:44:53.189459 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189460    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ec5d8d-023b-43b2-934e-df7706831ace-kserve-provision-location\") pod \"d4ec5d8d-023b-43b2-934e-df7706831ace\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") "
Sep 17 19:44:53.189748 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189485    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-rbac-proxy-sar-config\") pod \"d4ec5d8d-023b-43b2-934e-df7706831ace\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") "
Sep 17 19:44:53.189748 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189515    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ec5d8d-023b-43b2-934e-df7706831ace-proxy-tls\") pod \"d4ec5d8d-023b-43b2-934e-df7706831ace\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") "
Sep 17 19:44:53.189748 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v89xl\" (UniqueName: \"kubernetes.io/projected/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-api-access-v89xl\") pod \"d4ec5d8d-023b-43b2-934e-df7706831ace\" (UID: \"d4ec5d8d-023b-43b2-934e-df7706831ace\") "
Sep 17 19:44:53.189914 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189849    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4ec5d8d-023b-43b2-934e-df7706831ace" (UID: "d4ec5d8d-023b-43b2-934e-df7706831ace"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:44:53.189914 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189859    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ec5d8d-023b-43b2-934e-df7706831ace-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4ec5d8d-023b-43b2-934e-df7706831ace" (UID: "d4ec5d8d-023b-43b2-934e-df7706831ace"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:53.189914 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189894    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4ec5d8d-023b-43b2-934e-df7706831ace" (UID: "d4ec5d8d-023b-43b2-934e-df7706831ace"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:44:53.190062 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189978    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:53.190062 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.189996    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ec5d8d-023b-43b2-934e-df7706831ace-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:53.190062 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.190007    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:53.191825 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.191794    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-api-access-v89xl" (OuterVolumeSpecName: "kube-api-access-v89xl") pod "d4ec5d8d-023b-43b2-934e-df7706831ace" (UID: "d4ec5d8d-023b-43b2-934e-df7706831ace"). InnerVolumeSpecName "kube-api-access-v89xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:44:53.191825 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.191805    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ec5d8d-023b-43b2-934e-df7706831ace-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4ec5d8d-023b-43b2-934e-df7706831ace" (UID: "d4ec5d8d-023b-43b2-934e-df7706831ace"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:44:53.291112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.291025    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ec5d8d-023b-43b2-934e-df7706831ace-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:53.291112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:53.291057    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89xl\" (UniqueName: \"kubernetes.io/projected/d4ec5d8d-023b-43b2-934e-df7706831ace-kube-api-access-v89xl\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:54.030592 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:54.030561    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"
Sep 17 19:44:54.031084 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:54.030564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz" event={"ID":"d4ec5d8d-023b-43b2-934e-df7706831ace","Type":"ContainerDied","Data":"e663d2ec99e4ee561fe9f33b807ee1aa498acf3a82edb65db3d68bcde3f8e9cb"}
Sep 17 19:44:54.031084 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:54.030716    2583 scope.go:117] "RemoveContainer" containerID="7eecddb0df60f41c3580ec95cd805a558e931783a0d6f532a523d38a485a2056"
Sep 17 19:44:54.032194 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:54.032172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" event={"ID":"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528","Type":"ContainerStarted","Data":"7c4c64cc1713f96107f35958c1e6c23d399182a4ca2f9f71002de66bf922ed82"}
Sep 17 19:44:54.032294 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:54.032226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" event={"ID":"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528","Type":"ContainerStarted","Data":"70cbf6a0f0c9449b4ff826a256f51ea08d95568db8d3ae93fdb6e971ba5fc816"}
Sep 17 19:44:54.040319 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:54.040304    2583 scope.go:117] "RemoveContainer" containerID="97ba449df4da7ecd46ceb6c27582600f84d8794fa79c96ad6fefbd95a947f6f3"
Sep 17 19:44:54.048214 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:54.048192    2583 scope.go:117] "RemoveContainer" containerID="c4fd1a4f23924c7ee4cfa3263951236461d986ae81d6d0edade93553d8e972ec"
Sep 17 19:44:54.067768 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:54.067700    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"]
Sep 17 19:44:54.069608 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:54.069584    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-8jmrz"]
Sep 17 19:44:55.038457 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:55.038427    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" path="/var/lib/kubelet/pods/d4ec5d8d-023b-43b2-934e-df7706831ace/volumes"
Sep 17 19:44:55.834894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:55.834849    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Sep 17 19:44:55.840251 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:55.840227    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.58:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 19:44:57.048705 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:57.048668    2583 generic.go:356] "Generic (PLEG): container finished" podID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerID="7c4c64cc1713f96107f35958c1e6c23d399182a4ca2f9f71002de66bf922ed82" exitCode=0
Sep 17 19:44:57.049076 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:57.048749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" event={"ID":"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528","Type":"ContainerDied","Data":"7c4c64cc1713f96107f35958c1e6c23d399182a4ca2f9f71002de66bf922ed82"}
Sep 17 19:44:58.056132 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:58.056099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" event={"ID":"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528","Type":"ContainerStarted","Data":"e5a8805d54f1d11290d0b2823a67e33702568977f70c9930be16cce6ed5978ba"}
Sep 17 19:44:58.056597 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:58.056164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" event={"ID":"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528","Type":"ContainerStarted","Data":"e23720785ffe00625ae555aa205cfdd6f826e20889b05da54361b8121357a6cf"}
Sep 17 19:44:58.056597 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:58.056381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:58.076948 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:58.076899    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podStartSLOduration=2.252446382 podStartE2EDuration="6.076886148s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824439766s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:44:58.074770024 +0000 UTC m=+2317.202511426" watchObservedRunningTime="2026-09-17 19:44:58.076886148 +0000 UTC m=+2317.204627617"
Sep 17 19:44:59.063429 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.063397    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerID="3ac3d10da1269c32e45f064901c465ab81f1f855572fec99992699ce8d67fd56" exitCode=0
Sep 17 19:44:59.063921 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.063471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" event={"ID":"2ff8533a-bead-44d0-831d-205e5b5e888f","Type":"ContainerDied","Data":"3ac3d10da1269c32e45f064901c465ab81f1f855572fec99992699ce8d67fd56"}
Sep 17 19:44:59.063921 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.063760    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:44:59.064939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.064912    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:44:59.146546 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.146524    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:44:59.240453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.240367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-rbac-proxy-sar-config\") pod \"2ff8533a-bead-44d0-831d-205e5b5e888f\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") "
Sep 17 19:44:59.240453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.240437    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4ggz\" (UniqueName: \"kubernetes.io/projected/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-api-access-w4ggz\") pod \"2ff8533a-bead-44d0-831d-205e5b5e888f\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") "
Sep 17 19:44:59.240683 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.240631    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff8533a-bead-44d0-831d-205e5b5e888f-proxy-tls\") pod \"2ff8533a-bead-44d0-831d-205e5b5e888f\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") "
Sep 17 19:44:59.240747 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.240702    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff8533a-bead-44d0-831d-205e5b5e888f-kserve-provision-location\") pod \"2ff8533a-bead-44d0-831d-205e5b5e888f\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") "
Sep 17 19:44:59.240802 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.240758    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-cabundle-cert\") pod \"2ff8533a-bead-44d0-831d-205e5b5e888f\" (UID: \"2ff8533a-bead-44d0-831d-205e5b5e888f\") "
Sep 17 19:44:59.240853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.240790    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ff8533a-bead-44d0-831d-205e5b5e888f" (UID: "2ff8533a-bead-44d0-831d-205e5b5e888f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:44:59.241018 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.241002    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:59.241082 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.240962    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff8533a-bead-44d0-831d-205e5b5e888f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ff8533a-bead-44d0-831d-205e5b5e888f" (UID: "2ff8533a-bead-44d0-831d-205e5b5e888f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:59.241320 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.241296    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ff8533a-bead-44d0-831d-205e5b5e888f" (UID: "2ff8533a-bead-44d0-831d-205e5b5e888f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:44:59.242783 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.242761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff8533a-bead-44d0-831d-205e5b5e888f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ff8533a-bead-44d0-831d-205e5b5e888f" (UID: "2ff8533a-bead-44d0-831d-205e5b5e888f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:44:59.243432 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.243410    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-api-access-w4ggz" (OuterVolumeSpecName: "kube-api-access-w4ggz") pod "2ff8533a-bead-44d0-831d-205e5b5e888f" (UID: "2ff8533a-bead-44d0-831d-205e5b5e888f"). InnerVolumeSpecName "kube-api-access-w4ggz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:44:59.341943 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.341912    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ggz\" (UniqueName: \"kubernetes.io/projected/2ff8533a-bead-44d0-831d-205e5b5e888f-kube-api-access-w4ggz\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:59.341943 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.341942    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff8533a-bead-44d0-831d-205e5b5e888f-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:59.342148 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.341953    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff8533a-bead-44d0-831d-205e5b5e888f-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:44:59.342148 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:44:59.341962    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff8533a-bead-44d0-831d-205e5b5e888f-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:45:00.069747 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:00.069718    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"
Sep 17 19:45:00.069747 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:00.069738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6" event={"ID":"2ff8533a-bead-44d0-831d-205e5b5e888f","Type":"ContainerDied","Data":"7151f3ad9a49291788112e34a817aba78fc7ef6c76d84a57a1128214bddd6f7a"}
Sep 17 19:45:00.070309 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:00.069796    2583 scope.go:117] "RemoveContainer" containerID="564125b5d96ce6c80b36860df2db2544310b01b427a236616c2dabed9b075bd8"
Sep 17 19:45:00.070597 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:00.070554    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:45:00.079245 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:00.079228    2583 scope.go:117] "RemoveContainer" containerID="3ac3d10da1269c32e45f064901c465ab81f1f855572fec99992699ce8d67fd56"
Sep 17 19:45:00.087048 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:00.087029    2583 scope.go:117] "RemoveContainer" containerID="4c6077a157ac4c537201490020635db519fcded9f766b16e7edc3c5190bfb100"
Sep 17 19:45:00.092061 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:00.092039    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"]
Sep 17 19:45:00.095745 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:00.095722    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-f8mt6"]
Sep 17 19:45:01.044359 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:01.044315    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" path="/var/lib/kubelet/pods/2ff8533a-bead-44d0-831d-205e5b5e888f/volumes"
Sep 17 19:45:05.074462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:05.074388    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:45:05.075052 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:05.075026    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:45:15.075839 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:15.075795    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:45:23.990993 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:23.990957    2583 scope.go:117] "RemoveContainer" containerID="9aa931845810808d9c2af18f35da30a4899008107693ef11cf1b37fc52bed68e"
Sep 17 19:45:25.075081 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:25.075043    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:45:35.075730 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:35.075692    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:45:45.075843 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:45.075803    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:45:55.075657 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:45:55.075597    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:46:05.075872 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:05.075829    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:46:15.075474 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:15.075425    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:46:24.018598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:24.018560    2583 scope.go:117] "RemoveContainer" containerID="80da9ac30445f50140d6ed2f473b1c572c76ce3f1c811b598af9392df2a49a91"
Sep 17 19:46:24.026770 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:24.026749    2583 scope.go:117] "RemoveContainer" containerID="c99130512aba82986f3533c758eebff7764de75f7359fc627a1dd00bb134e605"
Sep 17 19:46:25.034179 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:25.034132    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:46:27.851725 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:27.851697    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:46:27.856844 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:27.856819    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:46:35.038573 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:35.038495    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:46:43.315340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315299    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"]
Sep 17 19:46:43.315829 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315812    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kserve-container"
Sep 17 19:46:43.315907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315832    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kserve-container"
Sep 17 19:46:43.315907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315852    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="storage-initializer"
Sep 17 19:46:43.315907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315861    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="storage-initializer"
Sep 17 19:46:43.315907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315873    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kube-rbac-proxy"
Sep 17 19:46:43.315907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315882    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kube-rbac-proxy"
Sep 17 19:46:43.315907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315899    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container"
Sep 17 19:46:43.315907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315907    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container"
Sep 17 19:46:43.316264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315921    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="storage-initializer"
Sep 17 19:46:43.316264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315930    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="storage-initializer"
Sep 17 19:46:43.316264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315958    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kube-rbac-proxy"
Sep 17 19:46:43.316264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.315966    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kube-rbac-proxy"
Sep 17 19:46:43.316264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.316054    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kserve-container"
Sep 17 19:46:43.316264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.316064    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kube-rbac-proxy"
Sep 17 19:46:43.316264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.316075    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ec5d8d-023b-43b2-934e-df7706831ace" containerName="kserve-container"
Sep 17 19:46:43.316264 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.316089    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff8533a-bead-44d0-831d-205e5b5e888f" containerName="kube-rbac-proxy"
Sep 17 19:46:43.320557 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.320536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.322946 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.322926    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 19:46:43.323060 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.323022    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:46:43.328943 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.328917    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"]
Sep 17 19:46:43.419362 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.419333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de747006-80d6-42f4-aadc-445f01d55897-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.419529 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.419373    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnf5v\" (UniqueName: \"kubernetes.io/projected/de747006-80d6-42f4-aadc-445f01d55897-kube-api-access-vnf5v\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.419529 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.419395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.419529 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.419449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.419688 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.419550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de747006-80d6-42f4-aadc-445f01d55897-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.520877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.520843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.521048 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.520883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.521048 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.520944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de747006-80d6-42f4-aadc-445f01d55897-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.521048 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.520993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de747006-80d6-42f4-aadc-445f01d55897-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.521048 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.521038    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnf5v\" (UniqueName: \"kubernetes.io/projected/de747006-80d6-42f4-aadc-445f01d55897-kube-api-access-vnf5v\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.521391 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.521364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de747006-80d6-42f4-aadc-445f01d55897-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.521569 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.521549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.521607 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.521575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.523493 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.523476    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de747006-80d6-42f4-aadc-445f01d55897-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.528734 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.528705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnf5v\" (UniqueName: \"kubernetes.io/projected/de747006-80d6-42f4-aadc-445f01d55897-kube-api-access-vnf5v\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.631635 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.631539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:43.904874 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:43.904847    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"]
Sep 17 19:46:43.907363 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:46:43.907331    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde747006_80d6_42f4_aadc_445f01d55897.slice/crio-783bedd2190003abbdbe1dc0a9c27641aede8c2bea2c2c97acdc17b611896b43 WatchSource:0}: Error finding container 783bedd2190003abbdbe1dc0a9c27641aede8c2bea2c2c97acdc17b611896b43: Status 404 returned error can't find the container with id 783bedd2190003abbdbe1dc0a9c27641aede8c2bea2c2c97acdc17b611896b43
Sep 17 19:46:44.158858 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:44.158776    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"]
Sep 17 19:46:44.159151 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:44.159094    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" containerID="cri-o://e23720785ffe00625ae555aa205cfdd6f826e20889b05da54361b8121357a6cf" gracePeriod=30
Sep 17 19:46:44.159388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:44.159175    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kube-rbac-proxy" containerID="cri-o://e5a8805d54f1d11290d0b2823a67e33702568977f70c9930be16cce6ed5978ba" gracePeriod=30
Sep 17 19:46:44.503287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:44.503248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" event={"ID":"de747006-80d6-42f4-aadc-445f01d55897","Type":"ContainerStarted","Data":"652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7"}
Sep 17 19:46:44.503742 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:44.503320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" event={"ID":"de747006-80d6-42f4-aadc-445f01d55897","Type":"ContainerStarted","Data":"783bedd2190003abbdbe1dc0a9c27641aede8c2bea2c2c97acdc17b611896b43"}
Sep 17 19:46:44.505553 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:44.505525    2583 generic.go:356] "Generic (PLEG): container finished" podID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerID="e5a8805d54f1d11290d0b2823a67e33702568977f70c9930be16cce6ed5978ba" exitCode=2
Sep 17 19:46:44.505669 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:44.505571    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" event={"ID":"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528","Type":"ContainerDied","Data":"e5a8805d54f1d11290d0b2823a67e33702568977f70c9930be16cce6ed5978ba"}
Sep 17 19:46:45.034573 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:45.034525    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 19:46:45.070927 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:45.070893    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 17 19:46:48.523990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:48.523958    2583 generic.go:356] "Generic (PLEG): container finished" podID="de747006-80d6-42f4-aadc-445f01d55897" containerID="652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7" exitCode=0
Sep 17 19:46:48.524492 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:48.524030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" event={"ID":"de747006-80d6-42f4-aadc-445f01d55897","Type":"ContainerDied","Data":"652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7"}
Sep 17 19:46:49.324520 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.324487    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"]
Sep 17 19:46:49.327880 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.327864    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.330160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.330138    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:46:49.330249 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.330138    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 19:46:49.346797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.346768    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"]
Sep 17 19:46:49.473772 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.473735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f86048ee-a007-4bbb-ab6b-5566144d83fb-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.473772 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.473772    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.473969 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.473790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t96h\" (UniqueName: \"kubernetes.io/projected/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-api-access-5t96h\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.473969 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.473897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.473969 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.473932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f86048ee-a007-4bbb-ab6b-5566144d83fb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.530433 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.530398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" event={"ID":"de747006-80d6-42f4-aadc-445f01d55897","Type":"ContainerStarted","Data":"de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8"}
Sep 17 19:46:49.530817 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.530457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" event={"ID":"de747006-80d6-42f4-aadc-445f01d55897","Type":"ContainerStarted","Data":"f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7"}
Sep 17 19:46:49.530817 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.530675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:49.530912 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.530846    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:49.550073 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.550024    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" podStartSLOduration=2.538116998 podStartE2EDuration="6.55000774s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.011890742s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:46:49.549202089 +0000 UTC m=+2428.676943493" watchObservedRunningTime="2026-09-17 19:46:49.55000774 +0000 UTC m=+2428.677749141"
Sep 17 19:46:49.575065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.574983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.575065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.575025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f86048ee-a007-4bbb-ab6b-5566144d83fb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.575263 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.575077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f86048ee-a007-4bbb-ab6b-5566144d83fb-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.575263 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.575094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.575263 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.575111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t96h\" (UniqueName: \"kubernetes.io/projected/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-api-access-5t96h\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.575419 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:46:49.575336    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 17 19:46:49.575419 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:46:49.575415    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f86048ee-a007-4bbb-ab6b-5566144d83fb-proxy-tls podName:f86048ee-a007-4bbb-ab6b-5566144d83fb nodeName:}" failed. No retries permitted until 2026-09-17 19:46:50.075393162 +0000 UTC m=+2429.203134559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f86048ee-a007-4bbb-ab6b-5566144d83fb-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" (UID: "f86048ee-a007-4bbb-ab6b-5566144d83fb") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 17 19:46:49.575559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.575527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f86048ee-a007-4bbb-ab6b-5566144d83fb-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.575873 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.575851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.575873 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.575861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:49.583179 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:49.583158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t96h\" (UniqueName: \"kubernetes.io/projected/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-api-access-5t96h\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:50.070635 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.070586    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 17 19:46:50.079810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.079786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f86048ee-a007-4bbb-ab6b-5566144d83fb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:50.082166 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.082138    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f86048ee-a007-4bbb-ab6b-5566144d83fb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:50.238667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.238629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:50.537434 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.537395    2583 generic.go:356] "Generic (PLEG): container finished" podID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerID="e23720785ffe00625ae555aa205cfdd6f826e20889b05da54361b8121357a6cf" exitCode=0
Sep 17 19:46:50.537940 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.537472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" event={"ID":"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528","Type":"ContainerDied","Data":"e23720785ffe00625ae555aa205cfdd6f826e20889b05da54361b8121357a6cf"}
Sep 17 19:46:50.569976 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.569939    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"]
Sep 17 19:46:50.571593 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:46:50.571570    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86048ee_a007_4bbb_ab6b_5566144d83fb.slice/crio-1d3f112bfe6b0e9807de17a4cb187d07b1159337b92cb5c29f6a0f86d63add3c WatchSource:0}: Error finding container 1d3f112bfe6b0e9807de17a4cb187d07b1159337b92cb5c29f6a0f86d63add3c: Status 404 returned error can't find the container with id 1d3f112bfe6b0e9807de17a4cb187d07b1159337b92cb5c29f6a0f86d63add3c
Sep 17 19:46:50.701827 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.701803    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:46:50.887773 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.887682    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-cabundle-cert\") pod \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") "
Sep 17 19:46:50.887773 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.887734    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-proxy-tls\") pod \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") "
Sep 17 19:46:50.887773 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.887766    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kserve-provision-location\") pod \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") "
Sep 17 19:46:50.888019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.887816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvgv5\" (UniqueName: \"kubernetes.io/projected/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-api-access-bvgv5\") pod \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") "
Sep 17 19:46:50.888019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.887850    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-rbac-proxy-sar-config\") pod \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\" (UID: \"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528\") "
Sep 17 19:46:50.888132 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.888086    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" (UID: "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:46:50.888132 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.888094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" (UID: "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:50.888324 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.888301    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" (UID: "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:46:50.890059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.890031    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" (UID: "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:46:50.890157 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.890057    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-api-access-bvgv5" (OuterVolumeSpecName: "kube-api-access-bvgv5") pod "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" (UID: "2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528"). InnerVolumeSpecName "kube-api-access-bvgv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:46:50.988940 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.988905    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:46:50.988940 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.988936    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:46:50.988940 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.988945    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:46:50.989158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.988954    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvgv5\" (UniqueName: \"kubernetes.io/projected/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-api-access-bvgv5\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:46:50.989158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:50.988963    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:46:51.543569 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:51.543535    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq" event={"ID":"2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528","Type":"ContainerDied","Data":"70cbf6a0f0c9449b4ff826a256f51ea08d95568db8d3ae93fdb6e971ba5fc816"}
Sep 17 19:46:51.543569 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:51.543559    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"
Sep 17 19:46:51.543982 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:51.543600    2583 scope.go:117] "RemoveContainer" containerID="e5a8805d54f1d11290d0b2823a67e33702568977f70c9930be16cce6ed5978ba"
Sep 17 19:46:51.546096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:51.546064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" event={"ID":"f86048ee-a007-4bbb-ab6b-5566144d83fb","Type":"ContainerStarted","Data":"b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa"}
Sep 17 19:46:51.546229 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:51.546128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" event={"ID":"f86048ee-a007-4bbb-ab6b-5566144d83fb","Type":"ContainerStarted","Data":"1d3f112bfe6b0e9807de17a4cb187d07b1159337b92cb5c29f6a0f86d63add3c"}
Sep 17 19:46:51.553400 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:51.553384    2583 scope.go:117] "RemoveContainer" containerID="e23720785ffe00625ae555aa205cfdd6f826e20889b05da54361b8121357a6cf"
Sep 17 19:46:51.560786 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:51.560767    2583 scope.go:117] "RemoveContainer" containerID="7c4c64cc1713f96107f35958c1e6c23d399182a4ca2f9f71002de66bf922ed82"
Sep 17 19:46:51.580575 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:51.580551    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"]
Sep 17 19:46:51.584206 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:51.584169    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2gtq"]
Sep 17 19:46:53.038628 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:53.038594    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" path="/var/lib/kubelet/pods/2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528/volumes"
Sep 17 19:46:54.561993 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:54.561902    2583 generic.go:356] "Generic (PLEG): container finished" podID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerID="b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa" exitCode=0
Sep 17 19:46:54.561993 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:54.561977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" event={"ID":"f86048ee-a007-4bbb-ab6b-5566144d83fb","Type":"ContainerDied","Data":"b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa"}
Sep 17 19:46:55.543994 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:55.543966    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:46:55.569792 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:55.569759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" event={"ID":"f86048ee-a007-4bbb-ab6b-5566144d83fb","Type":"ContainerStarted","Data":"85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca"}
Sep 17 19:46:55.570191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:55.569815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" event={"ID":"f86048ee-a007-4bbb-ab6b-5566144d83fb","Type":"ContainerStarted","Data":"f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85"}
Sep 17 19:46:55.570191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:55.570022    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:55.570191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:55.570165    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:46:55.592043 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:46:55.591982    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" podStartSLOduration=2.9757638 podStartE2EDuration="6.591963287s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.616199487s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:46:55.591702337 +0000 UTC m=+2434.719443741" watchObservedRunningTime="2026-09-17 19:46:55.591963287 +0000 UTC m=+2434.719704689"
Sep 17 19:47:01.580983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:01.580954    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:47:25.547951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:25.547911    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 19:47:31.584938 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:31.584911    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:47:35.546335 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:35.546307    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:47:39.380399 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:39.380363    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"]
Sep 17 19:47:39.380894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:39.380589    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="kserve-container" containerID="cri-o://f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85" gracePeriod=30
Sep 17 19:47:39.380894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:39.380666    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="kube-rbac-proxy" containerID="cri-o://85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca" gracePeriod=30
Sep 17 19:47:39.767467 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:39.767372    2583 generic.go:356] "Generic (PLEG): container finished" podID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerID="85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca" exitCode=2
Sep 17 19:47:39.767467 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:39.767450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" event={"ID":"f86048ee-a007-4bbb-ab6b-5566144d83fb","Type":"ContainerDied","Data":"85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca"}
Sep 17 19:47:40.324703 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.324676    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:47:40.505779 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.505687    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f86048ee-a007-4bbb-ab6b-5566144d83fb-proxy-tls\") pod \"f86048ee-a007-4bbb-ab6b-5566144d83fb\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") "
Sep 17 19:47:40.505779 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.505756    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f86048ee-a007-4bbb-ab6b-5566144d83fb-kserve-provision-location\") pod \"f86048ee-a007-4bbb-ab6b-5566144d83fb\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") "
Sep 17 19:47:40.505779 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.505774    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-cabundle-cert\") pod \"f86048ee-a007-4bbb-ab6b-5566144d83fb\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") "
Sep 17 19:47:40.506333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.505828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-rbac-proxy-sar-config\") pod \"f86048ee-a007-4bbb-ab6b-5566144d83fb\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") "
Sep 17 19:47:40.506333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.505873    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t96h\" (UniqueName: \"kubernetes.io/projected/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-api-access-5t96h\") pod \"f86048ee-a007-4bbb-ab6b-5566144d83fb\" (UID: \"f86048ee-a007-4bbb-ab6b-5566144d83fb\") "
Sep 17 19:47:40.506333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.506105    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86048ee-a007-4bbb-ab6b-5566144d83fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f86048ee-a007-4bbb-ab6b-5566144d83fb" (UID: "f86048ee-a007-4bbb-ab6b-5566144d83fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:47:40.506333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.506140    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f86048ee-a007-4bbb-ab6b-5566144d83fb" (UID: "f86048ee-a007-4bbb-ab6b-5566144d83fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:47:40.506333 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.506170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f86048ee-a007-4bbb-ab6b-5566144d83fb" (UID: "f86048ee-a007-4bbb-ab6b-5566144d83fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:47:40.508094 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.508071    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f86048ee-a007-4bbb-ab6b-5566144d83fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f86048ee-a007-4bbb-ab6b-5566144d83fb" (UID: "f86048ee-a007-4bbb-ab6b-5566144d83fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:47:40.508178 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.508150    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-api-access-5t96h" (OuterVolumeSpecName: "kube-api-access-5t96h") pod "f86048ee-a007-4bbb-ab6b-5566144d83fb" (UID: "f86048ee-a007-4bbb-ab6b-5566144d83fb"). InnerVolumeSpecName "kube-api-access-5t96h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:47:40.607355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.607313    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f86048ee-a007-4bbb-ab6b-5566144d83fb-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:40.607355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.607350    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:40.607355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.607362    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:40.607355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.607371    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t96h\" (UniqueName: \"kubernetes.io/projected/f86048ee-a007-4bbb-ab6b-5566144d83fb-kube-api-access-5t96h\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:40.607641 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.607379    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f86048ee-a007-4bbb-ab6b-5566144d83fb-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:40.773830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.773739    2583 generic.go:356] "Generic (PLEG): container finished" podID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerID="f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85" exitCode=0
Sep 17 19:47:40.773830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.773792    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"
Sep 17 19:47:40.773830 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.773820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" event={"ID":"f86048ee-a007-4bbb-ab6b-5566144d83fb","Type":"ContainerDied","Data":"f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85"}
Sep 17 19:47:40.774064 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.773877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z" event={"ID":"f86048ee-a007-4bbb-ab6b-5566144d83fb","Type":"ContainerDied","Data":"1d3f112bfe6b0e9807de17a4cb187d07b1159337b92cb5c29f6a0f86d63add3c"}
Sep 17 19:47:40.774064 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.773905    2583 scope.go:117] "RemoveContainer" containerID="85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca"
Sep 17 19:47:40.782832 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.782817    2583 scope.go:117] "RemoveContainer" containerID="f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85"
Sep 17 19:47:40.790503 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.790486    2583 scope.go:117] "RemoveContainer" containerID="b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa"
Sep 17 19:47:40.796965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.796942    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"]
Sep 17 19:47:40.799208 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.799189    2583 scope.go:117] "RemoveContainer" containerID="85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca"
Sep 17 19:47:40.799531 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:47:40.799514    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca\": container with ID starting with 85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca not found: ID does not exist" containerID="85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca"
Sep 17 19:47:40.799588 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.799538    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca"} err="failed to get container status \"85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca\": rpc error: code = NotFound desc = could not find container \"85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca\": container with ID starting with 85de096115766de8c68d21bc7f2a6fd15bd55caa55b7a070d8f0593432ec8fca not found: ID does not exist"
Sep 17 19:47:40.799588 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.799556    2583 scope.go:117] "RemoveContainer" containerID="f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85"
Sep 17 19:47:40.799714 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.799587    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-tg48z"]
Sep 17 19:47:40.799848 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:47:40.799831    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85\": container with ID starting with f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85 not found: ID does not exist" containerID="f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85"
Sep 17 19:47:40.799895 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.799855    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85"} err="failed to get container status \"f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85\": rpc error: code = NotFound desc = could not find container \"f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85\": container with ID starting with f799aaf7492f1c9a8af2e2bc84126238b399602d843d98a9e8525bfe540e5e85 not found: ID does not exist"
Sep 17 19:47:40.799895 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.799872    2583 scope.go:117] "RemoveContainer" containerID="b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa"
Sep 17 19:47:40.800142 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:47:40.800125    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa\": container with ID starting with b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa not found: ID does not exist" containerID="b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa"
Sep 17 19:47:40.800186 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:40.800147    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa"} err="failed to get container status \"b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa\": rpc error: code = NotFound desc = could not find container \"b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa\": container with ID starting with b00ae64a6804a6a15dbd0669ea4749c53deab5ae9e2e11e71947549b48dd25fa not found: ID does not exist"
Sep 17 19:47:41.039273 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:41.039194    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" path="/var/lib/kubelet/pods/f86048ee-a007-4bbb-ab6b-5566144d83fb/volumes"
Sep 17 19:47:43.371280 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:43.371249    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"]
Sep 17 19:47:43.371751 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:43.371549    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kserve-container" containerID="cri-o://f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7" gracePeriod=30
Sep 17 19:47:43.371751 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:43.371600    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kube-rbac-proxy" containerID="cri-o://de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8" gracePeriod=30
Sep 17 19:47:43.790511 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:43.790479    2583 generic.go:356] "Generic (PLEG): container finished" podID="de747006-80d6-42f4-aadc-445f01d55897" containerID="de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8" exitCode=2
Sep 17 19:47:43.790697 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:43.790547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" event={"ID":"de747006-80d6-42f4-aadc-445f01d55897","Type":"ContainerDied","Data":"de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8"}
Sep 17 19:47:44.574405 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574285    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"]
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574673    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="kube-rbac-proxy"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574687    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="kube-rbac-proxy"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574702    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574707    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574720    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kube-rbac-proxy"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574727    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kube-rbac-proxy"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574735    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="storage-initializer"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574740    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="storage-initializer"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574751    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="storage-initializer"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574756    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="storage-initializer"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574767    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="kserve-container"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574772    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="kserve-container"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574819    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kserve-container"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574826    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="kube-rbac-proxy"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574835    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86048ee-a007-4bbb-ab6b-5566144d83fb" containerName="kserve-container"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.574842    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9b9f9f-bc90-4ff8-87a5-8e3ce3ba0528" containerName="kube-rbac-proxy"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.578345    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.580508    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.580576    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 19:47:44.592059 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.589260    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"]
Sep 17 19:47:44.741262 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.741226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.741447 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.741278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f005989c-b644-4482-befb-d2a5b3bbeb10-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.741447 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.741361    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.741447 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.741394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f2hf\" (UniqueName: \"kubernetes.io/projected/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-api-access-6f2hf\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.741447 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.741420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f005989c-b644-4482-befb-d2a5b3bbeb10-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.842330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.842241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.842330 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.842285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f005989c-b644-4482-befb-d2a5b3bbeb10-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.842558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.842409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.842558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.842438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f2hf\" (UniqueName: \"kubernetes.io/projected/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-api-access-6f2hf\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.842558 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.842462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f005989c-b644-4482-befb-d2a5b3bbeb10-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.842767 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.842722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f005989c-b644-4482-befb-d2a5b3bbeb10-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.843179 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.843149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.843284 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.843207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.845143 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.845122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f005989c-b644-4482-befb-d2a5b3bbeb10-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.851406 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.851387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f2hf\" (UniqueName: \"kubernetes.io/projected/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-api-access-6f2hf\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:44.888423 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:44.888393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:45.173797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:45.173771    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"]
Sep 17 19:47:45.175882 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:47:45.175838    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf005989c_b644_4482_befb_d2a5b3bbeb10.slice/crio-3a62ac83ec4451f24712bc1d214472c637e80b858b7a79096670493a77bc36e7 WatchSource:0}: Error finding container 3a62ac83ec4451f24712bc1d214472c637e80b858b7a79096670493a77bc36e7: Status 404 returned error can't find the container with id 3a62ac83ec4451f24712bc1d214472c637e80b858b7a79096670493a77bc36e7
Sep 17 19:47:45.538504 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:45.538409    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 17 19:47:45.801513 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:45.801425    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" event={"ID":"f005989c-b644-4482-befb-d2a5b3bbeb10","Type":"ContainerStarted","Data":"b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061"}
Sep 17 19:47:45.801513 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:45.801484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" event={"ID":"f005989c-b644-4482-befb-d2a5b3bbeb10","Type":"ContainerStarted","Data":"3a62ac83ec4451f24712bc1d214472c637e80b858b7a79096670493a77bc36e7"}
Sep 17 19:47:46.586771 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:46.586726    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:47:49.819424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:49.819388    2583 generic.go:356] "Generic (PLEG): container finished" podID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerID="b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061" exitCode=0
Sep 17 19:47:49.819892 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:49.819449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" event={"ID":"f005989c-b644-4482-befb-d2a5b3bbeb10","Type":"ContainerDied","Data":"b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061"}
Sep 17 19:47:50.605293 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.605269    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:47:50.694957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.694868    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-kube-rbac-proxy-sar-config\") pod \"de747006-80d6-42f4-aadc-445f01d55897\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") "
Sep 17 19:47:50.694957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.694927    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de747006-80d6-42f4-aadc-445f01d55897-proxy-tls\") pod \"de747006-80d6-42f4-aadc-445f01d55897\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") "
Sep 17 19:47:50.694957 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.694957    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de747006-80d6-42f4-aadc-445f01d55897-kserve-provision-location\") pod \"de747006-80d6-42f4-aadc-445f01d55897\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") "
Sep 17 19:47:50.695237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.694975    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnf5v\" (UniqueName: \"kubernetes.io/projected/de747006-80d6-42f4-aadc-445f01d55897-kube-api-access-vnf5v\") pod \"de747006-80d6-42f4-aadc-445f01d55897\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") "
Sep 17 19:47:50.695237 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.695013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-cabundle-cert\") pod \"de747006-80d6-42f4-aadc-445f01d55897\" (UID: \"de747006-80d6-42f4-aadc-445f01d55897\") "
Sep 17 19:47:50.695352 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.695285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de747006-80d6-42f4-aadc-445f01d55897" (UID: "de747006-80d6-42f4-aadc-445f01d55897"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:47:50.695412 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.695355    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de747006-80d6-42f4-aadc-445f01d55897-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de747006-80d6-42f4-aadc-445f01d55897" (UID: "de747006-80d6-42f4-aadc-445f01d55897"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:47:50.695503 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.695465    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de747006-80d6-42f4-aadc-445f01d55897" (UID: "de747006-80d6-42f4-aadc-445f01d55897"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:47:50.697267 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.697245    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de747006-80d6-42f4-aadc-445f01d55897-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de747006-80d6-42f4-aadc-445f01d55897" (UID: "de747006-80d6-42f4-aadc-445f01d55897"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:47:50.697374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.697300    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de747006-80d6-42f4-aadc-445f01d55897-kube-api-access-vnf5v" (OuterVolumeSpecName: "kube-api-access-vnf5v") pod "de747006-80d6-42f4-aadc-445f01d55897" (UID: "de747006-80d6-42f4-aadc-445f01d55897"). InnerVolumeSpecName "kube-api-access-vnf5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:47:50.795954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.795911    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de747006-80d6-42f4-aadc-445f01d55897-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:50.795954 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.795954    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de747006-80d6-42f4-aadc-445f01d55897-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:50.796155 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.795971    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnf5v\" (UniqueName: \"kubernetes.io/projected/de747006-80d6-42f4-aadc-445f01d55897-kube-api-access-vnf5v\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:50.796155 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.795987    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:50.796155 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.796001    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de747006-80d6-42f4-aadc-445f01d55897-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:47:50.826900 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.826863    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" event={"ID":"f005989c-b644-4482-befb-d2a5b3bbeb10","Type":"ContainerStarted","Data":"4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f"}
Sep 17 19:47:50.827286 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.826950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" event={"ID":"f005989c-b644-4482-befb-d2a5b3bbeb10","Type":"ContainerStarted","Data":"43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6"}
Sep 17 19:47:50.827286 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.827179    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:50.828941 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.828912    2583 generic.go:356] "Generic (PLEG): container finished" podID="de747006-80d6-42f4-aadc-445f01d55897" containerID="f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7" exitCode=0
Sep 17 19:47:50.829047 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.828942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" event={"ID":"de747006-80d6-42f4-aadc-445f01d55897","Type":"ContainerDied","Data":"f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7"}
Sep 17 19:47:50.829047 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.828962    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"
Sep 17 19:47:50.829047 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.828990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" event={"ID":"de747006-80d6-42f4-aadc-445f01d55897","Type":"ContainerDied","Data":"783bedd2190003abbdbe1dc0a9c27641aede8c2bea2c2c97acdc17b611896b43"}
Sep 17 19:47:50.829047 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.829015    2583 scope.go:117] "RemoveContainer" containerID="de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8"
Sep 17 19:47:50.840258 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.840242    2583 scope.go:117] "RemoveContainer" containerID="f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7"
Sep 17 19:47:50.846852 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.846810    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" podStartSLOduration=3.112436077 podStartE2EDuration="6.846796057s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.73435998s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:47:50.845002422 +0000 UTC m=+2489.972743825" watchObservedRunningTime="2026-09-17 19:47:50.846796057 +0000 UTC m=+2489.974537458"
Sep 17 19:47:50.848357 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.848336    2583 scope.go:117] "RemoveContainer" containerID="652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7"
Sep 17 19:47:50.855973 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.855954    2583 scope.go:117] "RemoveContainer" containerID="de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8"
Sep 17 19:47:50.856360 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:47:50.856327    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8\": container with ID starting with de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8 not found: ID does not exist" containerID="de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8"
Sep 17 19:47:50.856475 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.856376    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8"} err="failed to get container status \"de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8\": rpc error: code = NotFound desc = could not find container \"de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8\": container with ID starting with de580133d8db114de07075e32fb60ef1a0cfec06e475ec862ec3cdccfbebc4f8 not found: ID does not exist"
Sep 17 19:47:50.856475 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.856404    2583 scope.go:117] "RemoveContainer" containerID="f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7"
Sep 17 19:47:50.856833 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:47:50.856782    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7\": container with ID starting with f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7 not found: ID does not exist" containerID="f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7"
Sep 17 19:47:50.856960 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.856841    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7"} err="failed to get container status \"f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7\": rpc error: code = NotFound desc = could not find container \"f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7\": container with ID starting with f5d538882706a388e107d275f272edb180cc582c3c2b0a82249513b2a90804f7 not found: ID does not exist"
Sep 17 19:47:50.856960 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.856864    2583 scope.go:117] "RemoveContainer" containerID="652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7"
Sep 17 19:47:50.857128 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:47:50.857108    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7\": container with ID starting with 652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7 not found: ID does not exist" containerID="652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7"
Sep 17 19:47:50.857217 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.857131    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7"} err="failed to get container status \"652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7\": rpc error: code = NotFound desc = could not find container \"652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7\": container with ID starting with 652b7f9d725865cd1195d1ea623c3546d5b9a4c181ab9294f5bb34d11adc87b7 not found: ID does not exist"
Sep 17 19:47:50.858213 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.858189    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"]
Sep 17 19:47:50.861795 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:50.861777    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8"]
Sep 17 19:47:51.039422 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:51.039338    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de747006-80d6-42f4-aadc-445f01d55897" path="/var/lib/kubelet/pods/de747006-80d6-42f4-aadc-445f01d55897/volumes"
Sep 17 19:47:51.538663 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:51.538593    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pbrd8" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": context deadline exceeded"
Sep 17 19:47:51.835737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:51.835648    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:51.837017 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:51.836993    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 19:47:52.840978 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:52.840936    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 19:47:57.845439 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:57.845411    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:47:57.845974 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:47:57.845946    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 19:48:07.846837 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:07.846758    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:48:14.684511 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:14.684480    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"]
Sep 17 19:48:14.684960 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:14.684755    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kserve-container" containerID="cri-o://43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6" gracePeriod=30
Sep 17 19:48:14.684960 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:14.684802    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kube-rbac-proxy" containerID="cri-o://4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f" gracePeriod=30
Sep 17 19:48:14.940220 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:14.940124    2583 generic.go:356] "Generic (PLEG): container finished" podID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerID="4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f" exitCode=2
Sep 17 19:48:14.940220 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:14.940190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" event={"ID":"f005989c-b644-4482-befb-d2a5b3bbeb10","Type":"ContainerDied","Data":"4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f"}
Sep 17 19:48:15.427346 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.427324    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:48:15.508914 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.508846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f005989c-b644-4482-befb-d2a5b3bbeb10-kserve-provision-location\") pod \"f005989c-b644-4482-befb-d2a5b3bbeb10\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") "
Sep 17 19:48:15.508914 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.508907    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-cabundle-cert\") pod \"f005989c-b644-4482-befb-d2a5b3bbeb10\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") "
Sep 17 19:48:15.509128 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.508957    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f005989c-b644-4482-befb-d2a5b3bbeb10-proxy-tls\") pod \"f005989c-b644-4482-befb-d2a5b3bbeb10\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") "
Sep 17 19:48:15.509128 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.508998    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f2hf\" (UniqueName: \"kubernetes.io/projected/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-api-access-6f2hf\") pod \"f005989c-b644-4482-befb-d2a5b3bbeb10\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") "
Sep 17 19:48:15.509128 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.509027    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-rbac-proxy-sar-config\") pod \"f005989c-b644-4482-befb-d2a5b3bbeb10\" (UID: \"f005989c-b644-4482-befb-d2a5b3bbeb10\") "
Sep 17 19:48:15.509300 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.509219    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f005989c-b644-4482-befb-d2a5b3bbeb10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f005989c-b644-4482-befb-d2a5b3bbeb10" (UID: "f005989c-b644-4482-befb-d2a5b3bbeb10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:48:15.509359 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.509301    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f005989c-b644-4482-befb-d2a5b3bbeb10-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:48:15.509359 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.509339    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f005989c-b644-4482-befb-d2a5b3bbeb10" (UID: "f005989c-b644-4482-befb-d2a5b3bbeb10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:48:15.509524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.509506    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f005989c-b644-4482-befb-d2a5b3bbeb10" (UID: "f005989c-b644-4482-befb-d2a5b3bbeb10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:48:15.511134 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.511116    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f005989c-b644-4482-befb-d2a5b3bbeb10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f005989c-b644-4482-befb-d2a5b3bbeb10" (UID: "f005989c-b644-4482-befb-d2a5b3bbeb10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:48:15.511335 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.511309    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-api-access-6f2hf" (OuterVolumeSpecName: "kube-api-access-6f2hf") pod "f005989c-b644-4482-befb-d2a5b3bbeb10" (UID: "f005989c-b644-4482-befb-d2a5b3bbeb10"). InnerVolumeSpecName "kube-api-access-6f2hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:48:15.609837 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.609810    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:48:15.609837 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.609834    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f005989c-b644-4482-befb-d2a5b3bbeb10-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:48:15.609837 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.609842    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f2hf\" (UniqueName: \"kubernetes.io/projected/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-api-access-6f2hf\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:48:15.610047 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.609851    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f005989c-b644-4482-befb-d2a5b3bbeb10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:48:15.946588 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.946551    2583 generic.go:356] "Generic (PLEG): container finished" podID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerID="43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6" exitCode=0
Sep 17 19:48:15.946994 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.946635    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"
Sep 17 19:48:15.946994 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.946642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" event={"ID":"f005989c-b644-4482-befb-d2a5b3bbeb10","Type":"ContainerDied","Data":"43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6"}
Sep 17 19:48:15.946994 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.946705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l" event={"ID":"f005989c-b644-4482-befb-d2a5b3bbeb10","Type":"ContainerDied","Data":"3a62ac83ec4451f24712bc1d214472c637e80b858b7a79096670493a77bc36e7"}
Sep 17 19:48:15.946994 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.946745    2583 scope.go:117] "RemoveContainer" containerID="4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f"
Sep 17 19:48:15.955334 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.955315    2583 scope.go:117] "RemoveContainer" containerID="43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6"
Sep 17 19:48:15.962806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.962790    2583 scope.go:117] "RemoveContainer" containerID="b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061"
Sep 17 19:48:15.968838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.968804    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"]
Sep 17 19:48:15.971143 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.971122    2583 scope.go:117] "RemoveContainer" containerID="4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f"
Sep 17 19:48:15.971397 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:48:15.971375    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f\": container with ID starting with 4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f not found: ID does not exist" containerID="4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f"
Sep 17 19:48:15.971465 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.971407    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f"} err="failed to get container status \"4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f\": rpc error: code = NotFound desc = could not find container \"4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f\": container with ID starting with 4042187d0c46cbc7740f6392b6441aff7130b1bef904f4ace36b68fd42a93f3f not found: ID does not exist"
Sep 17 19:48:15.971465 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.971424    2583 scope.go:117] "RemoveContainer" containerID="43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6"
Sep 17 19:48:15.971676 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:48:15.971655    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6\": container with ID starting with 43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6 not found: ID does not exist" containerID="43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6"
Sep 17 19:48:15.971737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.971682    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6"} err="failed to get container status \"43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6\": rpc error: code = NotFound desc = could not find container \"43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6\": container with ID starting with 43952edd6230a4798520af1e6413c4bf5ce01a51158dcb76c69aec0ee32128b6 not found: ID does not exist"
Sep 17 19:48:15.971737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.971697    2583 scope.go:117] "RemoveContainer" containerID="b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061"
Sep 17 19:48:15.971950 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:48:15.971931    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061\": container with ID starting with b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061 not found: ID does not exist" containerID="b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061"
Sep 17 19:48:15.972012 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.971954    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061"} err="failed to get container status \"b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061\": rpc error: code = NotFound desc = could not find container \"b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061\": container with ID starting with b30def3f44a27ddc0ca80aa2c279eec7f42be1baa26f0b62e28c5fc872bf6061 not found: ID does not exist"
Sep 17 19:48:15.972338 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:15.972318    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5rd7l"]
Sep 17 19:48:17.038158 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:17.038125    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" path="/var/lib/kubelet/pods/f005989c-b644-4482-befb-d2a5b3bbeb10/volumes"
Sep 17 19:48:19.860022 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.859981    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"]
Sep 17 19:48:19.860532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860452    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="storage-initializer"
Sep 17 19:48:19.860532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860472    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="storage-initializer"
Sep 17 19:48:19.860532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860484    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kserve-container"
Sep 17 19:48:19.860532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860491    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kserve-container"
Sep 17 19:48:19.860532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860505    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kserve-container"
Sep 17 19:48:19.860532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860512    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kserve-container"
Sep 17 19:48:19.860532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860529    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kube-rbac-proxy"
Sep 17 19:48:19.860532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860536    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kube-rbac-proxy"
Sep 17 19:48:19.860998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860551    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kube-rbac-proxy"
Sep 17 19:48:19.860998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860559    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kube-rbac-proxy"
Sep 17 19:48:19.860998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860569    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="storage-initializer"
Sep 17 19:48:19.860998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860578    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="storage-initializer"
Sep 17 19:48:19.860998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860673    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kserve-container"
Sep 17 19:48:19.860998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860688    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f005989c-b644-4482-befb-d2a5b3bbeb10" containerName="kube-rbac-proxy"
Sep 17 19:48:19.860998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860699    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kserve-container"
Sep 17 19:48:19.860998 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.860707    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de747006-80d6-42f4-aadc-445f01d55897" containerName="kube-rbac-proxy"
Sep 17 19:48:19.863420 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.863398    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:19.865676 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.865643    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 19:48:19.865800 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.865755    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:48:19.865917 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.865900    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:48:19.866534 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.866516    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:48:19.866534 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.866529    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:48:19.866744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.866587    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:48:19.866744 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.866600    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:48:19.875531 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.875507    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"]
Sep 17 19:48:19.943443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.943375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fac06ea-f21d-435f-856c-118e47b46ef6-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:19.943594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.943473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-api-access-s7rd6\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:19.943594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.943519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:19.943594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.943556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fac06ea-f21d-435f-856c-118e47b46ef6-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:19.943594 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:19.943588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.044827 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.044792    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fac06ea-f21d-435f-856c-118e47b46ef6-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.045026 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.044837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.045026 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.044857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fac06ea-f21d-435f-856c-118e47b46ef6-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.045026 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.044889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-api-access-s7rd6\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.045026 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.044913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.045389 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.045365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fac06ea-f21d-435f-856c-118e47b46ef6-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.045587 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.045568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.045648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.045570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.047287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.047269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fac06ea-f21d-435f-856c-118e47b46ef6-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.053866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.053845    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-api-access-s7rd6\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.172985 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.172957    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:20.446206 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.446181    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"]
Sep 17 19:48:20.448152 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:48:20.448120    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fac06ea_f21d_435f_856c_118e47b46ef6.slice/crio-df76de9d4b1a59893969451a021c6f64a46c6b2a21edb0e2123d28c56bb7adaa WatchSource:0}: Error finding container df76de9d4b1a59893969451a021c6f64a46c6b2a21edb0e2123d28c56bb7adaa: Status 404 returned error can't find the container with id df76de9d4b1a59893969451a021c6f64a46c6b2a21edb0e2123d28c56bb7adaa
Sep 17 19:48:20.975712 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.975673    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" event={"ID":"8fac06ea-f21d-435f-856c-118e47b46ef6","Type":"ContainerStarted","Data":"53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8"}
Sep 17 19:48:20.976173 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:20.975745    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" event={"ID":"8fac06ea-f21d-435f-856c-118e47b46ef6","Type":"ContainerStarted","Data":"df76de9d4b1a59893969451a021c6f64a46c6b2a21edb0e2123d28c56bb7adaa"}
Sep 17 19:48:24.994555 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:24.994524    2583 generic.go:356] "Generic (PLEG): container finished" podID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerID="53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8" exitCode=0
Sep 17 19:48:24.994879 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:24.994571    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" event={"ID":"8fac06ea-f21d-435f-856c-118e47b46ef6","Type":"ContainerDied","Data":"53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8"}
Sep 17 19:48:26.001717 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:26.001682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" event={"ID":"8fac06ea-f21d-435f-856c-118e47b46ef6","Type":"ContainerStarted","Data":"f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190"}
Sep 17 19:48:26.002104 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:26.001742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" event={"ID":"8fac06ea-f21d-435f-856c-118e47b46ef6","Type":"ContainerStarted","Data":"df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae"}
Sep 17 19:48:26.002104 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:26.001953    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:26.002207 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:26.002127    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:48:26.021256 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:26.021216    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" podStartSLOduration=2.66481902 podStartE2EDuration="7.021202407s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.356383387s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:48:26.019720455 +0000 UTC m=+2525.147461858" watchObservedRunningTime="2026-09-17 19:48:26.021202407 +0000 UTC m=+2525.148943840"
Sep 17 19:48:32.011797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:48:32.011759    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:49:02.016528 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:02.016493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:49:09.993387 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:09.993351    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"]
Sep 17 19:49:09.993945 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:09.993709    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="kserve-container" containerID="cri-o://df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae" gracePeriod=30
Sep 17 19:49:09.993945 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:09.993884    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="kube-rbac-proxy" containerID="cri-o://f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190" gracePeriod=30
Sep 17 19:49:10.184570 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:10.184532    2583 generic.go:356] "Generic (PLEG): container finished" podID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerID="f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190" exitCode=2
Sep 17 19:49:10.184780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:10.184603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" event={"ID":"8fac06ea-f21d-435f-856c-118e47b46ef6","Type":"ContainerDied","Data":"f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190"}
Sep 17 19:49:11.129421 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.129398    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:49:11.190570 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.190538    2583 generic.go:356] "Generic (PLEG): container finished" podID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerID="df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae" exitCode=0
Sep 17 19:49:11.190737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.190583    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" event={"ID":"8fac06ea-f21d-435f-856c-118e47b46ef6","Type":"ContainerDied","Data":"df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae"}
Sep 17 19:49:11.190737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.190588    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"
Sep 17 19:49:11.190737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.190660    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh" event={"ID":"8fac06ea-f21d-435f-856c-118e47b46ef6","Type":"ContainerDied","Data":"df76de9d4b1a59893969451a021c6f64a46c6b2a21edb0e2123d28c56bb7adaa"}
Sep 17 19:49:11.190737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.190717    2583 scope.go:117] "RemoveContainer" containerID="f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190"
Sep 17 19:49:11.198625 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.198599    2583 scope.go:117] "RemoveContainer" containerID="df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae"
Sep 17 19:49:11.206332 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.206313    2583 scope.go:117] "RemoveContainer" containerID="53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8"
Sep 17 19:49:11.213386 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.213368    2583 scope.go:117] "RemoveContainer" containerID="f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190"
Sep 17 19:49:11.213687 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:49:11.213661    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190\": container with ID starting with f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190 not found: ID does not exist" containerID="f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190"
Sep 17 19:49:11.213798 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.213691    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190"} err="failed to get container status \"f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190\": rpc error: code = NotFound desc = could not find container \"f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190\": container with ID starting with f988ac32295762a0fe33a4b9aeb0b261e1339f2ca3e080266614f538d2d33190 not found: ID does not exist"
Sep 17 19:49:11.213798 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.213707    2583 scope.go:117] "RemoveContainer" containerID="df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae"
Sep 17 19:49:11.213964 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:49:11.213949    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae\": container with ID starting with df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae not found: ID does not exist" containerID="df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae"
Sep 17 19:49:11.214003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.213970    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae"} err="failed to get container status \"df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae\": rpc error: code = NotFound desc = could not find container \"df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae\": container with ID starting with df21731386f2cb372a70fcc35ab99a4eaea6a570ee0786cd02cc9b6422fd10ae not found: ID does not exist"
Sep 17 19:49:11.214003 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.213988    2583 scope.go:117] "RemoveContainer" containerID="53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8"
Sep 17 19:49:11.214242 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:49:11.214224    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8\": container with ID starting with 53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8 not found: ID does not exist" containerID="53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8"
Sep 17 19:49:11.214278 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.214248    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8"} err="failed to get container status \"53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8\": rpc error: code = NotFound desc = could not find container \"53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8\": container with ID starting with 53172c1fc9c8bc7c89ea3a9a30e8c136592c09c79e634cfe6222638ba5470bb8 not found: ID does not exist"
Sep 17 19:49:11.242545 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.242519    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-rbac-proxy-sar-config\") pod \"8fac06ea-f21d-435f-856c-118e47b46ef6\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") "
Sep 17 19:49:11.242669 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.242577    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fac06ea-f21d-435f-856c-118e47b46ef6-proxy-tls\") pod \"8fac06ea-f21d-435f-856c-118e47b46ef6\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") "
Sep 17 19:49:11.242669 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.242652    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fac06ea-f21d-435f-856c-118e47b46ef6-kserve-provision-location\") pod \"8fac06ea-f21d-435f-856c-118e47b46ef6\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") "
Sep 17 19:49:11.242789 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.242679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-api-access-s7rd6\") pod \"8fac06ea-f21d-435f-856c-118e47b46ef6\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") "
Sep 17 19:49:11.242789 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.242722    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-cabundle-cert\") pod \"8fac06ea-f21d-435f-856c-118e47b46ef6\" (UID: \"8fac06ea-f21d-435f-856c-118e47b46ef6\") "
Sep 17 19:49:11.242942 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.242922    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8fac06ea-f21d-435f-856c-118e47b46ef6" (UID: "8fac06ea-f21d-435f-856c-118e47b46ef6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:49:11.243055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.243019    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fac06ea-f21d-435f-856c-118e47b46ef6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fac06ea-f21d-435f-856c-118e47b46ef6" (UID: "8fac06ea-f21d-435f-856c-118e47b46ef6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:11.243175 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.243146    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8fac06ea-f21d-435f-856c-118e47b46ef6" (UID: "8fac06ea-f21d-435f-856c-118e47b46ef6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:49:11.244672 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.244651    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fac06ea-f21d-435f-856c-118e47b46ef6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fac06ea-f21d-435f-856c-118e47b46ef6" (UID: "8fac06ea-f21d-435f-856c-118e47b46ef6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:49:11.244880 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.244860    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-api-access-s7rd6" (OuterVolumeSpecName: "kube-api-access-s7rd6") pod "8fac06ea-f21d-435f-856c-118e47b46ef6" (UID: "8fac06ea-f21d-435f-856c-118e47b46ef6"). InnerVolumeSpecName "kube-api-access-s7rd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:49:11.343523 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.343492    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fac06ea-f21d-435f-856c-118e47b46ef6-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:49:11.343523 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.343518    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fac06ea-f21d-435f-856c-118e47b46ef6-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:49:11.343523 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.343529    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-api-access-s7rd6\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:49:11.343743 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.343538    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:49:11.343743 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.343547    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fac06ea-f21d-435f-856c-118e47b46ef6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:49:11.511961 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.511933    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"]
Sep 17 19:49:11.515501 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:11.515479    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-twpqh"]
Sep 17 19:49:13.038008 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:13.037979    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" path="/var/lib/kubelet/pods/8fac06ea-f21d-435f-856c-118e47b46ef6/volumes"
Sep 17 19:49:14.318182 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.318146    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"]
Sep 17 19:49:14.318527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.318468    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="kube-rbac-proxy"
Sep 17 19:49:14.318527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.318478    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="kube-rbac-proxy"
Sep 17 19:49:14.318527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.318489    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="storage-initializer"
Sep 17 19:49:14.318527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.318494    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="storage-initializer"
Sep 17 19:49:14.318527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.318510    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="kserve-container"
Sep 17 19:49:14.318527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.318516    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="kserve-container"
Sep 17 19:49:14.318766 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.318558    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="kserve-container"
Sep 17 19:49:14.318766 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.318568    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fac06ea-f21d-435f-856c-118e47b46ef6" containerName="kube-rbac-proxy"
Sep 17 19:49:14.323182 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.323162    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.325570 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.325544    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 19:49:14.325570 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.325556    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 19:49:14.325570 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.325555    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:49:14.325884 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.325867    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:49:14.326292 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.326273    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:49:14.326407 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.326309    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:49:14.331281 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.331260    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"]
Sep 17 19:49:14.467554 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.467527    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.467701 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.467557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.467701 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.467579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.467701 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.467677    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr4jr\" (UniqueName: \"kubernetes.io/projected/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-api-access-kr4jr\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.568869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.568799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.568869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.568831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.568869 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.568850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.569086 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.568873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr4jr\" (UniqueName: \"kubernetes.io/projected/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-api-access-kr4jr\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.569215 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.569199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.569524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.569507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.571315 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.571298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.577341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.577317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr4jr\" (UniqueName: \"kubernetes.io/projected/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-api-access-kr4jr\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.634396 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.634378    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:14.910490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:14.910467    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"]
Sep 17 19:49:14.912938 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:49:14.912907    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73304cf_3ac1_41fd_8ac6_f958ffcd5a1d.slice/crio-3678f7a8123bbf7d71d0e5ff7fd562406929177894fb949e57af536f136125e8 WatchSource:0}: Error finding container 3678f7a8123bbf7d71d0e5ff7fd562406929177894fb949e57af536f136125e8: Status 404 returned error can't find the container with id 3678f7a8123bbf7d71d0e5ff7fd562406929177894fb949e57af536f136125e8
Sep 17 19:49:15.143006 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.142973    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"]
Sep 17 19:49:15.146846 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.146826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.149272 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.149244    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:49:15.149272 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.149245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 19:49:15.149464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.149256    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 19:49:15.149464 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.149353    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:49:15.156836 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.156815    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"]
Sep 17 19:49:15.210422 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.210384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" event={"ID":"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d","Type":"ContainerStarted","Data":"b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb"}
Sep 17 19:49:15.210659 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.210442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" event={"ID":"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d","Type":"ContainerStarted","Data":"3678f7a8123bbf7d71d0e5ff7fd562406929177894fb949e57af536f136125e8"}
Sep 17 19:49:15.277199 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.277165    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.277400 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.277227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c758dba-d49b-4772-9c46-1159e7b87154-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.277400 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.277255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jgk\" (UniqueName: \"kubernetes.io/projected/1c758dba-d49b-4772-9c46-1159e7b87154-kube-api-access-v9jgk\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.277400 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.277338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.277567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.277421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c758dba-d49b-4772-9c46-1159e7b87154-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.378401 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.378370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c758dba-d49b-4772-9c46-1159e7b87154-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.378401 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.378404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jgk\" (UniqueName: \"kubernetes.io/projected/1c758dba-d49b-4772-9c46-1159e7b87154-kube-api-access-v9jgk\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.378901 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.378435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.378901 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.378465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c758dba-d49b-4772-9c46-1159e7b87154-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.378901 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.378503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.378901 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.378868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c758dba-d49b-4772-9c46-1159e7b87154-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.379189 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.379167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.379228 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.379185    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.381087 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.381071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c758dba-d49b-4772-9c46-1159e7b87154-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.387207 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.387188    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jgk\" (UniqueName: \"kubernetes.io/projected/1c758dba-d49b-4772-9c46-1159e7b87154-kube-api-access-v9jgk\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-4vb26\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.457311 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.457279    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:15.734098 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:15.734072    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"]
Sep 17 19:49:15.735848 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:49:15.735818    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c758dba_d49b_4772_9c46_1159e7b87154.slice/crio-6432f0804c951f69cb14cf85f63f536a01bd8bf9d460305d40aa62326b1ff163 WatchSource:0}: Error finding container 6432f0804c951f69cb14cf85f63f536a01bd8bf9d460305d40aa62326b1ff163: Status 404 returned error can't find the container with id 6432f0804c951f69cb14cf85f63f536a01bd8bf9d460305d40aa62326b1ff163
Sep 17 19:49:16.216733 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:16.216681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerStarted","Data":"128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33"}
Sep 17 19:49:16.216936 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:16.216816    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerStarted","Data":"6432f0804c951f69cb14cf85f63f536a01bd8bf9d460305d40aa62326b1ff163"}
Sep 17 19:49:17.222598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:17.222568    2583 generic.go:356] "Generic (PLEG): container finished" podID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerID="b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb" exitCode=0
Sep 17 19:49:17.222961 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:17.222646    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" event={"ID":"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d","Type":"ContainerDied","Data":"b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb"}
Sep 17 19:49:18.228714 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:18.228679    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" event={"ID":"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d","Type":"ContainerStarted","Data":"4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556"}
Sep 17 19:49:18.229094 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:18.228767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" event={"ID":"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d","Type":"ContainerStarted","Data":"473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8"}
Sep 17 19:49:18.229094 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:18.228947    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:18.248004 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:18.247949    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podStartSLOduration=2.98451192 podStartE2EDuration="4.247931838s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.263419918s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:49:18.247077026 +0000 UTC m=+2577.374818426" watchObservedRunningTime="2026-09-17 19:49:18.247931838 +0000 UTC m=+2577.375673240"
Sep 17 19:49:19.233066 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:19.233038    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:19.234348 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:19.234309    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:49:20.238221 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:20.238190    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c758dba-d49b-4772-9c46-1159e7b87154" containerID="128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33" exitCode=0
Sep 17 19:49:20.238690 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:20.238266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerDied","Data":"128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33"}
Sep 17 19:49:20.238690 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:20.238630    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:49:20.300475 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:20.300457    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:49:21.247261 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:21.247210    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerStarted","Data":"649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf"}
Sep 17 19:49:23.260361 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:23.260320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerStarted","Data":"bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547"}
Sep 17 19:49:23.260858 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:23.260589    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:23.260858 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:23.260664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:23.260858 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:23.260690    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerStarted","Data":"150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c"}
Sep 17 19:49:23.284057 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:23.284010    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podStartSLOduration=1.7731863429999999 podStartE2EDuration="8.28399674s" totalImagesPullingTime="2.7909645s" totalInitContainerRuntime="3.719845897s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:49:23.282694089 +0000 UTC m=+2582.410435490" watchObservedRunningTime="2026-09-17 19:49:23.28399674 +0000 UTC m=+2582.411738142"
Sep 17 19:49:24.266519 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:24.266478    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:25.243280 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:25.243249    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:49:25.244009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:25.243975    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:49:30.275703 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:30.275677    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:49:35.244723 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:35.244639    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:49:45.244205 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:45.244166    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:49:50.277256 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:50.277217    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 19:49:55.244275 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:55.244235    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:49:55.457891 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:49:55.457835    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 19:50:00.453257 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:00.453228    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:50:05.244016 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:05.243965    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:50:15.244209 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:15.244169    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:50:25.244555 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:25.244515    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:50:30.455089 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:30.455058    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:50:35.158537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:35.158495    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"]
Sep 17 19:50:35.158980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:35.158784    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-container" containerID="cri-o://649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf" gracePeriod=30
Sep 17 19:50:35.158980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:35.158830    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-agent" containerID="cri-o://150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c" gracePeriod=30
Sep 17 19:50:35.158980 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:35.158827    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy" containerID="cri-o://bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547" gracePeriod=30
Sep 17 19:50:35.244508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:35.244489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:50:35.271233 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:35.271205    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 17 19:50:35.574160 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:35.574129    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c758dba-d49b-4772-9c46-1159e7b87154" containerID="bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547" exitCode=2
Sep 17 19:50:35.574329 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:35.574201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerDied","Data":"bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547"}
Sep 17 19:50:37.589359 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:37.589276    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c758dba-d49b-4772-9c46-1159e7b87154" containerID="649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf" exitCode=0
Sep 17 19:50:37.589359 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:37.589334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerDied","Data":"649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf"}
Sep 17 19:50:40.271185 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.271147    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 17 19:50:40.276628 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.276584    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 19:50:40.365379 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.365349    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"]
Sep 17 19:50:40.368735 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.368709    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.370877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.370854    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 19:50:40.370877 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.370871    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 19:50:40.379945 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.379922    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"]
Sep 17 19:50:40.487934 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.487904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvnm\" (UniqueName: \"kubernetes.io/projected/336be974-5ffc-4e48-ade4-d07a399436d3-kube-api-access-hwvnm\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.487934 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.487945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336be974-5ffc-4e48-ade4-d07a399436d3-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.488161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.487983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.488161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.488034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.488161 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.488071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336be974-5ffc-4e48-ade4-d07a399436d3-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.589392 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.589317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336be974-5ffc-4e48-ade4-d07a399436d3-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.589392 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.589359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvnm\" (UniqueName: \"kubernetes.io/projected/336be974-5ffc-4e48-ade4-d07a399436d3-kube-api-access-hwvnm\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.589392 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.589386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336be974-5ffc-4e48-ade4-d07a399436d3-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.589669 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.589423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.589669 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.589447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.589669 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:50:40.589565    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 19:50:40.589669 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:50:40.589662    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/336be974-5ffc-4e48-ade4-d07a399436d3-proxy-tls podName:336be974-5ffc-4e48-ade4-d07a399436d3 nodeName:}" failed. No retries permitted until 2026-09-17 19:50:41.08963985 +0000 UTC m=+2660.217381249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/336be974-5ffc-4e48-ade4-d07a399436d3-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-jdtqq" (UID: "336be974-5ffc-4e48-ade4-d07a399436d3") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 19:50:40.589884 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.589819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336be974-5ffc-4e48-ade4-d07a399436d3-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.590077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.590060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.590117 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.590086    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:40.601353 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:40.601334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvnm\" (UniqueName: \"kubernetes.io/projected/336be974-5ffc-4e48-ade4-d07a399436d3-kube-api-access-hwvnm\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:41.093318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:41.093286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336be974-5ffc-4e48-ade4-d07a399436d3-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:41.095828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:41.095800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336be974-5ffc-4e48-ade4-d07a399436d3-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-jdtqq\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:41.279075 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:41.279045    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:41.549907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:41.549872    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"]
Sep 17 19:50:41.553873 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:50:41.553832    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336be974_5ffc_4e48_ade4_d07a399436d3.slice/crio-9bbad4fb0ad29b80748f936aefbe17fd59c3275a261c58bea660d00099d33ad9 WatchSource:0}: Error finding container 9bbad4fb0ad29b80748f936aefbe17fd59c3275a261c58bea660d00099d33ad9: Status 404 returned error can't find the container with id 9bbad4fb0ad29b80748f936aefbe17fd59c3275a261c58bea660d00099d33ad9
Sep 17 19:50:41.610355 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:41.610321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" event={"ID":"336be974-5ffc-4e48-ade4-d07a399436d3","Type":"ContainerStarted","Data":"9bbad4fb0ad29b80748f936aefbe17fd59c3275a261c58bea660d00099d33ad9"}
Sep 17 19:50:42.616863 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:42.616828    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" event={"ID":"336be974-5ffc-4e48-ade4-d07a399436d3","Type":"ContainerStarted","Data":"42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439"}
Sep 17 19:50:44.875605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:44.875569    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"]
Sep 17 19:50:44.875978 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:44.875851    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" containerID="cri-o://473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8" gracePeriod=30
Sep 17 19:50:44.875978 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:44.875940    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kube-rbac-proxy" containerID="cri-o://4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556" gracePeriod=30
Sep 17 19:50:45.239506 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:45.239466    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 17 19:50:45.244655 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:45.244627    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 19:50:45.271567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:45.271537    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 17 19:50:45.271695 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:45.271681    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:50:45.633737 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:45.633657    2583 generic.go:356] "Generic (PLEG): container finished" podID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerID="4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556" exitCode=2
Sep 17 19:50:45.633883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:45.633736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" event={"ID":"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d","Type":"ContainerDied","Data":"4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556"}
Sep 17 19:50:47.644444 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:47.644404    2583 generic.go:356] "Generic (PLEG): container finished" podID="336be974-5ffc-4e48-ade4-d07a399436d3" containerID="42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439" exitCode=0
Sep 17 19:50:47.644864 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:47.644467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" event={"ID":"336be974-5ffc-4e48-ade4-d07a399436d3","Type":"ContainerDied","Data":"42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439"}
Sep 17 19:50:50.240798 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.240767    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:50:50.271202 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.271167    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 17 19:50:50.276720 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.276690    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 19:50:50.369538 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.369436    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kserve-provision-location\") pod \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") "
Sep 17 19:50:50.369538 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.369494    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr4jr\" (UniqueName: \"kubernetes.io/projected/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-api-access-kr4jr\") pod \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") "
Sep 17 19:50:50.369538 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.369539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-rbac-proxy-sar-config\") pod \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") "
Sep 17 19:50:50.369859 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.369583    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-proxy-tls\") pod \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\" (UID: \"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d\") "
Sep 17 19:50:50.369859 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.369837    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" (UID: "c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:50:50.369971 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.369946    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" (UID: "c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:50:50.372232 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.372194    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-api-access-kr4jr" (OuterVolumeSpecName: "kube-api-access-kr4jr") pod "c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" (UID: "c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d"). InnerVolumeSpecName "kube-api-access-kr4jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:50:50.372379 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.372231    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" (UID: "c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:50:50.470547 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.470508    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:50:50.470547 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.470537    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr4jr\" (UniqueName: \"kubernetes.io/projected/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-api-access-kr4jr\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:50:50.470547 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.470546    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:50:50.470547 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.470557    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:50:50.661645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.661540    2583 generic.go:356] "Generic (PLEG): container finished" podID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerID="473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8" exitCode=0
Sep 17 19:50:50.661645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.661600    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" event={"ID":"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d","Type":"ContainerDied","Data":"473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8"}
Sep 17 19:50:50.661853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.661679    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"
Sep 17 19:50:50.661853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.661712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw" event={"ID":"c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d","Type":"ContainerDied","Data":"3678f7a8123bbf7d71d0e5ff7fd562406929177894fb949e57af536f136125e8"}
Sep 17 19:50:50.661853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.661741    2583 scope.go:117] "RemoveContainer" containerID="4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556"
Sep 17 19:50:50.671789 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.671770    2583 scope.go:117] "RemoveContainer" containerID="473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8"
Sep 17 19:50:50.680631 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.680589    2583 scope.go:117] "RemoveContainer" containerID="b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb"
Sep 17 19:50:50.688104 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.688080    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"]
Sep 17 19:50:50.689975 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.689954    2583 scope.go:117] "RemoveContainer" containerID="4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556"
Sep 17 19:50:50.690274 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:50:50.690250    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556\": container with ID starting with 4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556 not found: ID does not exist" containerID="4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556"
Sep 17 19:50:50.690418 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.690317    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556"} err="failed to get container status \"4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556\": rpc error: code = NotFound desc = could not find container \"4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556\": container with ID starting with 4dbd7442430b99af925026a2eb877dd335d82795b58c5fae83ff2364c2b36556 not found: ID does not exist"
Sep 17 19:50:50.690418 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.690363    2583 scope.go:117] "RemoveContainer" containerID="473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8"
Sep 17 19:50:50.690965 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:50:50.690941    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8\": container with ID starting with 473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8 not found: ID does not exist" containerID="473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8"
Sep 17 19:50:50.691074 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.690982    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8"} err="failed to get container status \"473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8\": rpc error: code = NotFound desc = could not find container \"473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8\": container with ID starting with 473cd1ef87259e0291f7f40fb27f34dca7691d8ae07189bb8e44798ff23ee7e8 not found: ID does not exist"
Sep 17 19:50:50.691074 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.691007    2583 scope.go:117] "RemoveContainer" containerID="b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb"
Sep 17 19:50:50.691354 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:50:50.691318    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb\": container with ID starting with b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb not found: ID does not exist" containerID="b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb"
Sep 17 19:50:50.691452 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.691355    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb"} err="failed to get container status \"b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb\": rpc error: code = NotFound desc = could not find container \"b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb\": container with ID starting with b8a7eb4d9097997df20f9e4c57ca7d0f4d1bcb84d823cd1067b0923624470bbb not found: ID does not exist"
Sep 17 19:50:50.692650 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:50.692628    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-5wfqw"]
Sep 17 19:50:51.040065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:51.040030    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" path="/var/lib/kubelet/pods/c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d/volumes"
Sep 17 19:50:55.270916 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:55.270875    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 17 19:50:59.707714 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:59.707680    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" event={"ID":"336be974-5ffc-4e48-ade4-d07a399436d3","Type":"ContainerStarted","Data":"9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d"}
Sep 17 19:50:59.708118 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:59.707742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" event={"ID":"336be974-5ffc-4e48-ade4-d07a399436d3","Type":"ContainerStarted","Data":"025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49"}
Sep 17 19:50:59.708118 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:59.708026    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:59.708225 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:59.708198    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:50:59.709408 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:59.709384    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 19:50:59.727685 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:50:59.727643    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podStartSLOduration=3.276133519 podStartE2EDuration="19.727603727s" totalImagesPullingTime="11.283134486s" totalInitContainerRuntime="5.168335722s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:50:59.727105913 +0000 UTC m=+2678.854847315" watchObservedRunningTime="2026-09-17 19:50:59.727603727 +0000 UTC m=+2678.855345127"
Sep 17 19:51:00.270951 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:00.270908    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 17 19:51:00.276383 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:00.276356    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 19:51:00.276515 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:00.276472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:51:00.712634 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:00.712580    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 19:51:01.716382 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:01.716342    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 19:51:05.300345 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.300314    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:51:05.405555 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.405510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c758dba-d49b-4772-9c46-1159e7b87154-kserve-provision-location\") pod \"1c758dba-d49b-4772-9c46-1159e7b87154\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") "
Sep 17 19:51:05.405555 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.405558    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-kube-rbac-proxy-sar-config\") pod \"1c758dba-d49b-4772-9c46-1159e7b87154\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") "
Sep 17 19:51:05.405844 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.405595    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-cabundle-cert\") pod \"1c758dba-d49b-4772-9c46-1159e7b87154\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") "
Sep 17 19:51:05.405844 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.405636    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9jgk\" (UniqueName: \"kubernetes.io/projected/1c758dba-d49b-4772-9c46-1159e7b87154-kube-api-access-v9jgk\") pod \"1c758dba-d49b-4772-9c46-1159e7b87154\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") "
Sep 17 19:51:05.405844 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.405716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c758dba-d49b-4772-9c46-1159e7b87154-proxy-tls\") pod \"1c758dba-d49b-4772-9c46-1159e7b87154\" (UID: \"1c758dba-d49b-4772-9c46-1159e7b87154\") "
Sep 17 19:51:05.405997 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.405876    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c758dba-d49b-4772-9c46-1159e7b87154-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c758dba-d49b-4772-9c46-1159e7b87154" (UID: "1c758dba-d49b-4772-9c46-1159e7b87154"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:05.406086 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.406002    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c758dba-d49b-4772-9c46-1159e7b87154" (UID: "1c758dba-d49b-4772-9c46-1159e7b87154"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:51:05.406086 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.406015    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c758dba-d49b-4772-9c46-1159e7b87154" (UID: "1c758dba-d49b-4772-9c46-1159e7b87154"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:51:05.407982 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.407949    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c758dba-d49b-4772-9c46-1159e7b87154-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c758dba-d49b-4772-9c46-1159e7b87154" (UID: "1c758dba-d49b-4772-9c46-1159e7b87154"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:51:05.407982 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.407975    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c758dba-d49b-4772-9c46-1159e7b87154-kube-api-access-v9jgk" (OuterVolumeSpecName: "kube-api-access-v9jgk") pod "1c758dba-d49b-4772-9c46-1159e7b87154" (UID: "1c758dba-d49b-4772-9c46-1159e7b87154"). InnerVolumeSpecName "kube-api-access-v9jgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:51:05.506405 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.506372    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9jgk\" (UniqueName: \"kubernetes.io/projected/1c758dba-d49b-4772-9c46-1159e7b87154-kube-api-access-v9jgk\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:05.506405 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.506406    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c758dba-d49b-4772-9c46-1159e7b87154-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:05.506598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.506416    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c758dba-d49b-4772-9c46-1159e7b87154-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:05.506598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.506425    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:05.506598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.506434    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c758dba-d49b-4772-9c46-1159e7b87154-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:05.735326 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.735289    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c758dba-d49b-4772-9c46-1159e7b87154" containerID="150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c" exitCode=0
Sep 17 19:51:05.735493 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.735354    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"
Sep 17 19:51:05.735493 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.735375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerDied","Data":"150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c"}
Sep 17 19:51:05.735493 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.735450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" event={"ID":"1c758dba-d49b-4772-9c46-1159e7b87154","Type":"ContainerDied","Data":"6432f0804c951f69cb14cf85f63f536a01bd8bf9d460305d40aa62326b1ff163"}
Sep 17 19:51:05.735493 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.735483    2583 scope.go:117] "RemoveContainer" containerID="bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547"
Sep 17 19:51:05.745172 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.745153    2583 scope.go:117] "RemoveContainer" containerID="150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c"
Sep 17 19:51:05.754248 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.754229    2583 scope.go:117] "RemoveContainer" containerID="649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf"
Sep 17 19:51:05.757955 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.757906    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"]
Sep 17 19:51:05.761898 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.761878    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26"]
Sep 17 19:51:05.763479 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.763466    2583 scope.go:117] "RemoveContainer" containerID="128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33"
Sep 17 19:51:05.770657 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.770636    2583 scope.go:117] "RemoveContainer" containerID="bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547"
Sep 17 19:51:05.770933 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:51:05.770912    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547\": container with ID starting with bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547 not found: ID does not exist" containerID="bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547"
Sep 17 19:51:05.771022 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.770944    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547"} err="failed to get container status \"bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547\": rpc error: code = NotFound desc = could not find container \"bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547\": container with ID starting with bf94d31f1c8a6290b0bba3b3d0fdf35421ebe043120b262dbadb2240d08be547 not found: ID does not exist"
Sep 17 19:51:05.771022 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.770967    2583 scope.go:117] "RemoveContainer" containerID="150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c"
Sep 17 19:51:05.771195 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:51:05.771179    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c\": container with ID starting with 150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c not found: ID does not exist" containerID="150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c"
Sep 17 19:51:05.771241 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.771200    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c"} err="failed to get container status \"150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c\": rpc error: code = NotFound desc = could not find container \"150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c\": container with ID starting with 150b60315c4c3cbc331a4cd1f6f5d75c39c8593c1f264d27c2710e815a4ef22c not found: ID does not exist"
Sep 17 19:51:05.771241 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.771213    2583 scope.go:117] "RemoveContainer" containerID="649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf"
Sep 17 19:51:05.771454 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:51:05.771434    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf\": container with ID starting with 649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf not found: ID does not exist" containerID="649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf"
Sep 17 19:51:05.771517 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.771462    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf"} err="failed to get container status \"649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf\": rpc error: code = NotFound desc = could not find container \"649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf\": container with ID starting with 649202d909aa84b1f7b9b46842fb44ee2e06649d1d9825df673bb482d16698cf not found: ID does not exist"
Sep 17 19:51:05.771517 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.771485    2583 scope.go:117] "RemoveContainer" containerID="128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33"
Sep 17 19:51:05.771735 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:51:05.771711    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33\": container with ID starting with 128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33 not found: ID does not exist" containerID="128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33"
Sep 17 19:51:05.771817 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:05.771734    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33"} err="failed to get container status \"128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33\": rpc error: code = NotFound desc = could not find container \"128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33\": container with ID starting with 128266da8226a38e2ea9d15e65ede61bb71ed41f6bf1aa0bfe746eb2f4654a33 not found: ID does not exist"
Sep 17 19:51:06.272119 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:06.272077    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-4vb26" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:51:06.720528 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:06.720505    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:51:06.721049 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:06.721026    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 19:51:07.038878 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:07.038795    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" path="/var/lib/kubelet/pods/1c758dba-d49b-4772-9c46-1159e7b87154/volumes"
Sep 17 19:51:16.721170 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:16.721134    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 19:51:26.721328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:26.721287    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 19:51:27.882543 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:27.882511    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:51:27.888055 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:27.888037    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:51:36.721876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:36.721827    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 19:51:46.722259 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:46.722231    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:51:51.777933 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:51.777893    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"]
Sep 17 19:51:51.778318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:51.778142    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container" containerID="cri-o://025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49" gracePeriod=30
Sep 17 19:51:51.778318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:51.778166    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kube-rbac-proxy" containerID="cri-o://9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d" gracePeriod=30
Sep 17 19:51:51.928983 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:51.928951    2583 generic.go:356] "Generic (PLEG): container finished" podID="336be974-5ffc-4e48-ade4-d07a399436d3" containerID="9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d" exitCode=2
Sep 17 19:51:51.929129 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:51.929023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" event={"ID":"336be974-5ffc-4e48-ade4-d07a399436d3","Type":"ContainerDied","Data":"9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d"}
Sep 17 19:51:54.520846 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.520822    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:51:54.617210 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.617142    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwvnm\" (UniqueName: \"kubernetes.io/projected/336be974-5ffc-4e48-ade4-d07a399436d3-kube-api-access-hwvnm\") pod \"336be974-5ffc-4e48-ade4-d07a399436d3\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") "
Sep 17 19:51:54.617210 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.617189    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-cabundle-cert\") pod \"336be974-5ffc-4e48-ade4-d07a399436d3\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") "
Sep 17 19:51:54.617435 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.617215    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-kube-rbac-proxy-sar-config\") pod \"336be974-5ffc-4e48-ade4-d07a399436d3\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") "
Sep 17 19:51:54.617435 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.617286    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336be974-5ffc-4e48-ade4-d07a399436d3-kserve-provision-location\") pod \"336be974-5ffc-4e48-ade4-d07a399436d3\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") "
Sep 17 19:51:54.617435 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.617323    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336be974-5ffc-4e48-ade4-d07a399436d3-proxy-tls\") pod \"336be974-5ffc-4e48-ade4-d07a399436d3\" (UID: \"336be974-5ffc-4e48-ade4-d07a399436d3\") "
Sep 17 19:51:54.617599 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.617559    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "336be974-5ffc-4e48-ade4-d07a399436d3" (UID: "336be974-5ffc-4e48-ade4-d07a399436d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:51:54.617599 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.617574    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "336be974-5ffc-4e48-ade4-d07a399436d3" (UID: "336be974-5ffc-4e48-ade4-d07a399436d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:51:54.619530 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.619499    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336be974-5ffc-4e48-ade4-d07a399436d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "336be974-5ffc-4e48-ade4-d07a399436d3" (UID: "336be974-5ffc-4e48-ade4-d07a399436d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:51:54.619814 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.619786    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336be974-5ffc-4e48-ade4-d07a399436d3-kube-api-access-hwvnm" (OuterVolumeSpecName: "kube-api-access-hwvnm") pod "336be974-5ffc-4e48-ade4-d07a399436d3" (UID: "336be974-5ffc-4e48-ade4-d07a399436d3"). InnerVolumeSpecName "kube-api-access-hwvnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:51:54.626543 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.626514    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336be974-5ffc-4e48-ade4-d07a399436d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "336be974-5ffc-4e48-ade4-d07a399436d3" (UID: "336be974-5ffc-4e48-ade4-d07a399436d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:54.718853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.718830    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:54.718853 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.718850    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336be974-5ffc-4e48-ade4-d07a399436d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:54.718977 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.718860    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336be974-5ffc-4e48-ade4-d07a399436d3-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:54.718977 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.718870    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336be974-5ffc-4e48-ade4-d07a399436d3-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:54.718977 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.718878    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvnm\" (UniqueName: \"kubernetes.io/projected/336be974-5ffc-4e48-ade4-d07a399436d3-kube-api-access-hwvnm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:51:54.943590 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.943553    2583 generic.go:356] "Generic (PLEG): container finished" podID="336be974-5ffc-4e48-ade4-d07a399436d3" containerID="025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49" exitCode=0
Sep 17 19:51:54.943777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.943606    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"
Sep 17 19:51:54.943777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.943636    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" event={"ID":"336be974-5ffc-4e48-ade4-d07a399436d3","Type":"ContainerDied","Data":"025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49"}
Sep 17 19:51:54.943777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.943696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq" event={"ID":"336be974-5ffc-4e48-ade4-d07a399436d3","Type":"ContainerDied","Data":"9bbad4fb0ad29b80748f936aefbe17fd59c3275a261c58bea660d00099d33ad9"}
Sep 17 19:51:54.943777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.943722    2583 scope.go:117] "RemoveContainer" containerID="9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d"
Sep 17 19:51:54.952492 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.952475    2583 scope.go:117] "RemoveContainer" containerID="025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49"
Sep 17 19:51:54.961680 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.961660    2583 scope.go:117] "RemoveContainer" containerID="42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439"
Sep 17 19:51:54.966564 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.966538    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"]
Sep 17 19:51:54.970014 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.969994    2583 scope.go:117] "RemoveContainer" containerID="9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d"
Sep 17 19:51:54.970369 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:51:54.970334    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d\": container with ID starting with 9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d not found: ID does not exist" containerID="9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d"
Sep 17 19:51:54.970425 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.970382    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d"} err="failed to get container status \"9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d\": rpc error: code = NotFound desc = could not find container \"9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d\": container with ID starting with 9cebd8ef6b86a17395cfeec5d62a553278bbd7deda9d1a0ec2ad2ee32a8fc73d not found: ID does not exist"
Sep 17 19:51:54.970425 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.970408    2583 scope.go:117] "RemoveContainer" containerID="025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49"
Sep 17 19:51:54.970738 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:51:54.970718    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49\": container with ID starting with 025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49 not found: ID does not exist" containerID="025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49"
Sep 17 19:51:54.970823 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.970745    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49"} err="failed to get container status \"025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49\": rpc error: code = NotFound desc = could not find container \"025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49\": container with ID starting with 025b658e79611a93393dfb0cc7b08ad9ad445c02e58e112308b23636f43b3d49 not found: ID does not exist"
Sep 17 19:51:54.970823 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.970764    2583 scope.go:117] "RemoveContainer" containerID="42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439"
Sep 17 19:51:54.971118 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:51:54.971096    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439\": container with ID starting with 42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439 not found: ID does not exist" containerID="42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439"
Sep 17 19:51:54.971190 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.971126    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439"} err="failed to get container status \"42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439\": rpc error: code = NotFound desc = could not find container \"42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439\": container with ID starting with 42b655240098dc8176a5d7d3cae82338a9ed5817e0261327c7d25c0b23af6439 not found: ID does not exist"
Sep 17 19:51:54.971190 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:54.971176    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-jdtqq"]
Sep 17 19:51:55.038978 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:55.038948    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" path="/var/lib/kubelet/pods/336be974-5ffc-4e48-ade4-d07a399436d3/volumes"
Sep 17 19:51:56.960891 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.960863    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"]
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961167    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961178    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961190    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="storage-initializer"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961196    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="storage-initializer"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961204    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kube-rbac-proxy"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961209    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kube-rbac-proxy"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961221    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-container"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961227    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-container"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961234    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961239    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961245    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-agent"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961249    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-agent"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961254    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961260    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961267    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="storage-initializer"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961271    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="storage-initializer"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961278    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kube-rbac-proxy"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961282    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kube-rbac-proxy"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961294    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="storage-initializer"
Sep 17 19:51:56.961289 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961299    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="storage-initializer"
Sep 17 19:51:56.962030 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961345    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kube-rbac-proxy"
Sep 17 19:51:56.962030 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961353    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-container"
Sep 17 19:51:56.962030 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961358    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kserve-agent"
Sep 17 19:51:56.962030 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961364    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kube-rbac-proxy"
Sep 17 19:51:56.962030 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961369    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="336be974-5ffc-4e48-ade4-d07a399436d3" containerName="kserve-container"
Sep 17 19:51:56.962030 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961376    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c758dba-d49b-4772-9c46-1159e7b87154" containerName="kube-rbac-proxy"
Sep 17 19:51:56.962030 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.961381    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73304cf-3ac1-41fd-8ac6-f958ffcd5a1d" containerName="kserve-container"
Sep 17 19:51:56.965146 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.965130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:56.967518 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.967492    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:51:56.967689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.967604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:51:56.967689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.967606    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:51:56.967689 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.967651    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:51:56.968373 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.968353    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 19:51:56.968482 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.968385    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:51:56.968482 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.968434    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:51:56.975596 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:56.975571    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"]
Sep 17 19:51:57.037674 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.037603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.037866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.037719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3a7646-a623-4528-873b-e7dec98465e4-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.037866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.037795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlvgg\" (UniqueName: \"kubernetes.io/projected/8a3a7646-a623-4528-873b-e7dec98465e4-kube-api-access-tlvgg\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.037866 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.037828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7646-a623-4528-873b-e7dec98465e4-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.037990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.037869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.139047 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.139012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3a7646-a623-4528-873b-e7dec98465e4-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.139220 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.139090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlvgg\" (UniqueName: \"kubernetes.io/projected/8a3a7646-a623-4528-873b-e7dec98465e4-kube-api-access-tlvgg\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.139220 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.139122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7646-a623-4528-873b-e7dec98465e4-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.139220 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.139156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.139390 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.139282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.139644 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.139593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7646-a623-4528-873b-e7dec98465e4-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.139928 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.139907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.139928 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.139919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.141637 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.141604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3a7646-a623-4528-873b-e7dec98465e4-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.148307 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.148285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlvgg\" (UniqueName: \"kubernetes.io/projected/8a3a7646-a623-4528-873b-e7dec98465e4-kube-api-access-tlvgg\") pod \"isvc-paddle-runtime-predictor-58b99fff76-k8m7g\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.275810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.275740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:51:57.551203 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.551178    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"]
Sep 17 19:51:57.553341 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:51:57.553316    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3a7646_a623_4528_873b_e7dec98465e4.slice/crio-f16476aa87395295843ea71ff980da4ee6e4c11be6d15ba256faea3223696373 WatchSource:0}: Error finding container f16476aa87395295843ea71ff980da4ee6e4c11be6d15ba256faea3223696373: Status 404 returned error can't find the container with id f16476aa87395295843ea71ff980da4ee6e4c11be6d15ba256faea3223696373
Sep 17 19:51:57.959049 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.959011    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" event={"ID":"8a3a7646-a623-4528-873b-e7dec98465e4","Type":"ContainerStarted","Data":"eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500"}
Sep 17 19:51:57.959250 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:51:57.959076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" event={"ID":"8a3a7646-a623-4528-873b-e7dec98465e4","Type":"ContainerStarted","Data":"f16476aa87395295843ea71ff980da4ee6e4c11be6d15ba256faea3223696373"}
Sep 17 19:52:02.982302 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:02.982270    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a3a7646-a623-4528-873b-e7dec98465e4" containerID="eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500" exitCode=0
Sep 17 19:52:02.982741 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:02.982332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" event={"ID":"8a3a7646-a623-4528-873b-e7dec98465e4","Type":"ContainerDied","Data":"eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500"}
Sep 17 19:52:03.990477 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:03.990422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" event={"ID":"8a3a7646-a623-4528-873b-e7dec98465e4","Type":"ContainerStarted","Data":"e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f"}
Sep 17 19:52:03.990982 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:03.990547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" event={"ID":"8a3a7646-a623-4528-873b-e7dec98465e4","Type":"ContainerStarted","Data":"1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46"}
Sep 17 19:52:03.991559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:03.991523    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:52:03.991735 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:03.991633    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:52:03.993131 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:03.993102    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 19:52:04.023077 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:04.023029    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podStartSLOduration=3.453945972 podStartE2EDuration="8.023016395s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.569070423s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:52:04.022114856 +0000 UTC m=+2743.149856256" watchObservedRunningTime="2026-09-17 19:52:04.023016395 +0000 UTC m=+2743.150757795"
Sep 17 19:52:04.997196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:04.997151    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 19:52:06.176577 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.176541    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"]
Sep 17 19:52:06.179199 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.179184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.181383 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.181362    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 19:52:06.181531 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.181394    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:52:06.181531 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.181360    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 19:52:06.194235 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.194213    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"]
Sep 17 19:52:06.315826 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.315771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.315826 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.315826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81787dc-8e64-46a0-b26a-7b459074f2e1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.316066 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.315904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81787dc-8e64-46a0-b26a-7b459074f2e1-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.316066 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.315951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hbz\" (UniqueName: \"kubernetes.io/projected/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-api-access-l9hbz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.416317 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.416284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.416317 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.416317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81787dc-8e64-46a0-b26a-7b459074f2e1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.416543 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.416354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81787dc-8e64-46a0-b26a-7b459074f2e1-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.416543 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.416377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hbz\" (UniqueName: \"kubernetes.io/projected/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-api-access-l9hbz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.416543 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:52:06.416532    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 17 19:52:06.416664 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:52:06.416592    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81787dc-8e64-46a0-b26a-7b459074f2e1-proxy-tls podName:d81787dc-8e64-46a0-b26a-7b459074f2e1 nodeName:}" failed. No retries permitted until 2026-09-17 19:52:06.916576492 +0000 UTC m=+2746.044317872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d81787dc-8e64-46a0-b26a-7b459074f2e1-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b" (UID: "d81787dc-8e64-46a0-b26a-7b459074f2e1") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 17 19:52:06.416842 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.416816    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81787dc-8e64-46a0-b26a-7b459074f2e1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.417039 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.417020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.425460 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.425439    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hbz\" (UniqueName: \"kubernetes.io/projected/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-api-access-l9hbz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.920476 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.920439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81787dc-8e64-46a0-b26a-7b459074f2e1-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:06.923028 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:06.923008    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81787dc-8e64-46a0-b26a-7b459074f2e1-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:07.088571 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:07.088526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:07.361458 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:07.361431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"]
Sep 17 19:52:07.363984 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:52:07.363959    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81787dc_8e64_46a0_b26a_7b459074f2e1.slice/crio-ea6874a06948fd9f54dce47600e2fc8181bfa8cd9ce128663911e318e6efdc1d WatchSource:0}: Error finding container ea6874a06948fd9f54dce47600e2fc8181bfa8cd9ce128663911e318e6efdc1d: Status 404 returned error can't find the container with id ea6874a06948fd9f54dce47600e2fc8181bfa8cd9ce128663911e318e6efdc1d
Sep 17 19:52:08.013287 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:08.013252    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b" event={"ID":"d81787dc-8e64-46a0-b26a-7b459074f2e1","Type":"ContainerStarted","Data":"ea429d66bea3d0170205d079d3f0f901e77b05aa92827718634e5c6553483fc1"}
Sep 17 19:52:08.013541 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:08.013312    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b" event={"ID":"d81787dc-8e64-46a0-b26a-7b459074f2e1","Type":"ContainerStarted","Data":"ea6874a06948fd9f54dce47600e2fc8181bfa8cd9ce128663911e318e6efdc1d"}
Sep 17 19:52:10.002091 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:10.002064    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:52:10.002562 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:10.002536    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 19:52:11.028356 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:11.028281    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_d81787dc-8e64-46a0-b26a-7b459074f2e1/storage-initializer/0.log"
Sep 17 19:52:11.028356 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:11.028320    2583 generic.go:356] "Generic (PLEG): container finished" podID="d81787dc-8e64-46a0-b26a-7b459074f2e1" containerID="ea429d66bea3d0170205d079d3f0f901e77b05aa92827718634e5c6553483fc1" exitCode=1
Sep 17 19:52:11.028785 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:11.028375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b" event={"ID":"d81787dc-8e64-46a0-b26a-7b459074f2e1","Type":"ContainerDied","Data":"ea429d66bea3d0170205d079d3f0f901e77b05aa92827718634e5c6553483fc1"}
Sep 17 19:52:12.034672 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:12.034647    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_d81787dc-8e64-46a0-b26a-7b459074f2e1/storage-initializer/0.log"
Sep 17 19:52:12.035093 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:12.034725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b" event={"ID":"d81787dc-8e64-46a0-b26a-7b459074f2e1","Type":"ContainerStarted","Data":"9d3456c1b8e75d3a32bbc65b021e686b24b8861a537176bc92bc3b1302a8008b"}
Sep 17 19:52:15.047519 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:15.047494    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_d81787dc-8e64-46a0-b26a-7b459074f2e1/storage-initializer/1.log"
Sep 17 19:52:15.047975 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:15.047953    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_d81787dc-8e64-46a0-b26a-7b459074f2e1/storage-initializer/0.log"
Sep 17 19:52:15.048027 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:15.047983    2583 generic.go:356] "Generic (PLEG): container finished" podID="d81787dc-8e64-46a0-b26a-7b459074f2e1" containerID="9d3456c1b8e75d3a32bbc65b021e686b24b8861a537176bc92bc3b1302a8008b" exitCode=1
Sep 17 19:52:15.048092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:15.048063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b" event={"ID":"d81787dc-8e64-46a0-b26a-7b459074f2e1","Type":"ContainerDied","Data":"9d3456c1b8e75d3a32bbc65b021e686b24b8861a537176bc92bc3b1302a8008b"}
Sep 17 19:52:15.048166 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:15.048152    2583 scope.go:117] "RemoveContainer" containerID="ea429d66bea3d0170205d079d3f0f901e77b05aa92827718634e5c6553483fc1"
Sep 17 19:52:15.048560 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:15.048542    2583 scope.go:117] "RemoveContainer" containerID="ea429d66bea3d0170205d079d3f0f901e77b05aa92827718634e5c6553483fc1"
Sep 17 19:52:15.060001 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:52:15.059979    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_kserve-ci-e2e-test(d81787dc-8e64-46a0-b26a-7b459074f2e1)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b" podUID="d81787dc-8e64-46a0-b26a-7b459074f2e1"
Sep 17 19:52:15.060240 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:52:15.060211    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_kserve-ci-e2e-test_d81787dc-8e64-46a0-b26a-7b459074f2e1_0 in pod sandbox ea6874a06948fd9f54dce47600e2fc8181bfa8cd9ce128663911e318e6efdc1d from index: no such id: 'ea429d66bea3d0170205d079d3f0f901e77b05aa92827718634e5c6553483fc1'" containerID="ea429d66bea3d0170205d079d3f0f901e77b05aa92827718634e5c6553483fc1"
Sep 17 19:52:15.060294 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:15.060252    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea429d66bea3d0170205d079d3f0f901e77b05aa92827718634e5c6553483fc1"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_kserve-ci-e2e-test_d81787dc-8e64-46a0-b26a-7b459074f2e1_0 in pod sandbox ea6874a06948fd9f54dce47600e2fc8181bfa8cd9ce128663911e318e6efdc1d from index: no such id: 'ea429d66bea3d0170205d079d3f0f901e77b05aa92827718634e5c6553483fc1'"
Sep 17 19:52:16.053964 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.053934    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_d81787dc-8e64-46a0-b26a-7b459074f2e1/storage-initializer/1.log"
Sep 17 19:52:16.158932 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.158900    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"]
Sep 17 19:52:16.286501 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.286478    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_d81787dc-8e64-46a0-b26a-7b459074f2e1/storage-initializer/1.log"
Sep 17 19:52:16.287820 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.287805    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:16.301596 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.298851    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9hbz\" (UniqueName: \"kubernetes.io/projected/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-api-access-l9hbz\") pod \"d81787dc-8e64-46a0-b26a-7b459074f2e1\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") "
Sep 17 19:52:16.301596 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.298948    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-rbac-proxy-sar-config\") pod \"d81787dc-8e64-46a0-b26a-7b459074f2e1\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") "
Sep 17 19:52:16.301596 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.299006    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81787dc-8e64-46a0-b26a-7b459074f2e1-kserve-provision-location\") pod \"d81787dc-8e64-46a0-b26a-7b459074f2e1\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") "
Sep 17 19:52:16.301596 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.299038    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81787dc-8e64-46a0-b26a-7b459074f2e1-proxy-tls\") pod \"d81787dc-8e64-46a0-b26a-7b459074f2e1\" (UID: \"d81787dc-8e64-46a0-b26a-7b459074f2e1\") "
Sep 17 19:52:16.302500 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.302223    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81787dc-8e64-46a0-b26a-7b459074f2e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d81787dc-8e64-46a0-b26a-7b459074f2e1" (UID: "d81787dc-8e64-46a0-b26a-7b459074f2e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:52:16.304880 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.304716    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-api-access-l9hbz" (OuterVolumeSpecName: "kube-api-access-l9hbz") pod "d81787dc-8e64-46a0-b26a-7b459074f2e1" (UID: "d81787dc-8e64-46a0-b26a-7b459074f2e1"). InnerVolumeSpecName "kube-api-access-l9hbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:52:16.304970 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.304890    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81787dc-8e64-46a0-b26a-7b459074f2e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d81787dc-8e64-46a0-b26a-7b459074f2e1" (UID: "d81787dc-8e64-46a0-b26a-7b459074f2e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:52:16.305058 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.305034    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d81787dc-8e64-46a0-b26a-7b459074f2e1" (UID: "d81787dc-8e64-46a0-b26a-7b459074f2e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:52:16.400165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.400121    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:52:16.400165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.400161    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d81787dc-8e64-46a0-b26a-7b459074f2e1-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:52:16.400165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.400174    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81787dc-8e64-46a0-b26a-7b459074f2e1-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:52:16.400384 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:16.400190    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hbz\" (UniqueName: \"kubernetes.io/projected/d81787dc-8e64-46a0-b26a-7b459074f2e1-kube-api-access-l9hbz\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:52:17.060142 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:17.060112    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b_d81787dc-8e64-46a0-b26a-7b459074f2e1/storage-initializer/1.log"
Sep 17 19:52:17.060516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:17.060189    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b" event={"ID":"d81787dc-8e64-46a0-b26a-7b459074f2e1","Type":"ContainerDied","Data":"ea6874a06948fd9f54dce47600e2fc8181bfa8cd9ce128663911e318e6efdc1d"}
Sep 17 19:52:17.060516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:17.060215    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"
Sep 17 19:52:17.060516 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:17.060240    2583 scope.go:117] "RemoveContainer" containerID="9d3456c1b8e75d3a32bbc65b021e686b24b8861a537176bc92bc3b1302a8008b"
Sep 17 19:52:17.094041 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:17.094013    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"]
Sep 17 19:52:17.096119 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:17.096092    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-kcx6b"]
Sep 17 19:52:19.037873 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:19.037843    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81787dc-8e64-46a0-b26a-7b459074f2e1" path="/var/lib/kubelet/pods/d81787dc-8e64-46a0-b26a-7b459074f2e1/volumes"
Sep 17 19:52:20.003025 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:20.002984    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 19:52:30.002683 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:30.002639    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 19:52:40.003247 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:40.003204    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 19:52:50.003449 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:50.003417    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:52:58.365883 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:58.365843    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"]
Sep 17 19:52:58.366432 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:58.366100    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container" containerID="cri-o://1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46" gracePeriod=30
Sep 17 19:52:58.366432 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:58.366108    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kube-rbac-proxy" containerID="cri-o://e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f" gracePeriod=30
Sep 17 19:52:59.242493 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:59.242455    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a3a7646-a623-4528-873b-e7dec98465e4" containerID="e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f" exitCode=2
Sep 17 19:52:59.242709 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:59.242529    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" event={"ID":"8a3a7646-a623-4528-873b-e7dec98465e4","Type":"ContainerDied","Data":"e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f"}
Sep 17 19:52:59.997713 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:52:59.997676    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Sep 17 19:53:00.002843 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:00.002819    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 19:53:01.209797 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.209775    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:53:01.253790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.253759    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a3a7646-a623-4528-873b-e7dec98465e4" containerID="1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46" exitCode=0
Sep 17 19:53:01.253929 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.253810    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"
Sep 17 19:53:01.253929 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.253824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" event={"ID":"8a3a7646-a623-4528-873b-e7dec98465e4","Type":"ContainerDied","Data":"1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46"}
Sep 17 19:53:01.253929 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.253908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g" event={"ID":"8a3a7646-a623-4528-873b-e7dec98465e4","Type":"ContainerDied","Data":"f16476aa87395295843ea71ff980da4ee6e4c11be6d15ba256faea3223696373"}
Sep 17 19:53:01.254099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.253944    2583 scope.go:117] "RemoveContainer" containerID="e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f"
Sep 17 19:53:01.263807 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.263787    2583 scope.go:117] "RemoveContainer" containerID="1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46"
Sep 17 19:53:01.271406 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.271385    2583 scope.go:117] "RemoveContainer" containerID="eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500"
Sep 17 19:53:01.278828 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.278809    2583 scope.go:117] "RemoveContainer" containerID="e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f"
Sep 17 19:53:01.279111 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:53:01.279091    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f\": container with ID starting with e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f not found: ID does not exist" containerID="e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f"
Sep 17 19:53:01.279162 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.279120    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f"} err="failed to get container status \"e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f\": rpc error: code = NotFound desc = could not find container \"e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f\": container with ID starting with e8c298fe75c40bc7e0c67adea693aae231b7c826f338e893209a877f602c303f not found: ID does not exist"
Sep 17 19:53:01.279162 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.279139    2583 scope.go:117] "RemoveContainer" containerID="1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46"
Sep 17 19:53:01.279407 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:53:01.279389    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46\": container with ID starting with 1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46 not found: ID does not exist" containerID="1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46"
Sep 17 19:53:01.279471 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.279413    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46"} err="failed to get container status \"1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46\": rpc error: code = NotFound desc = could not find container \"1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46\": container with ID starting with 1ee8fb39120f29ea68cdffc7a55b525e34d158ec992653b1e409a05ee86fea46 not found: ID does not exist"
Sep 17 19:53:01.279471 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.279429    2583 scope.go:117] "RemoveContainer" containerID="eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500"
Sep 17 19:53:01.279740 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:53:01.279718    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500\": container with ID starting with eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500 not found: ID does not exist" containerID="eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500"
Sep 17 19:53:01.279806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.279749    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500"} err="failed to get container status \"eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500\": rpc error: code = NotFound desc = could not find container \"eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500\": container with ID starting with eaa159e8759bcb1b0f9ccae3ed7963bc94a540dd44946293bb9ea068cae71500 not found: ID does not exist"
Sep 17 19:53:01.385803 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.385739    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-cabundle-cert\") pod \"8a3a7646-a623-4528-873b-e7dec98465e4\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") "
Sep 17 19:53:01.385803 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.385779    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlvgg\" (UniqueName: \"kubernetes.io/projected/8a3a7646-a623-4528-873b-e7dec98465e4-kube-api-access-tlvgg\") pod \"8a3a7646-a623-4528-873b-e7dec98465e4\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") "
Sep 17 19:53:01.385955 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.385841    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-kube-rbac-proxy-sar-config\") pod \"8a3a7646-a623-4528-873b-e7dec98465e4\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") "
Sep 17 19:53:01.385955 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.385872    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3a7646-a623-4528-873b-e7dec98465e4-proxy-tls\") pod \"8a3a7646-a623-4528-873b-e7dec98465e4\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") "
Sep 17 19:53:01.385955 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.385906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7646-a623-4528-873b-e7dec98465e4-kserve-provision-location\") pod \"8a3a7646-a623-4528-873b-e7dec98465e4\" (UID: \"8a3a7646-a623-4528-873b-e7dec98465e4\") "
Sep 17 19:53:01.386145 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.386121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a3a7646-a623-4528-873b-e7dec98465e4" (UID: "8a3a7646-a623-4528-873b-e7dec98465e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:53:01.386251 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.386227    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a3a7646-a623-4528-873b-e7dec98465e4" (UID: "8a3a7646-a623-4528-873b-e7dec98465e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:53:01.388062 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.388041    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3a7646-a623-4528-873b-e7dec98465e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a3a7646-a623-4528-873b-e7dec98465e4" (UID: "8a3a7646-a623-4528-873b-e7dec98465e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:53:01.388196 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.388177    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3a7646-a623-4528-873b-e7dec98465e4-kube-api-access-tlvgg" (OuterVolumeSpecName: "kube-api-access-tlvgg") pod "8a3a7646-a623-4528-873b-e7dec98465e4" (UID: "8a3a7646-a623-4528-873b-e7dec98465e4"). InnerVolumeSpecName "kube-api-access-tlvgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:53:01.395817 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.395792    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3a7646-a623-4528-873b-e7dec98465e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a3a7646-a623-4528-873b-e7dec98465e4" (UID: "8a3a7646-a623-4528-873b-e7dec98465e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:53:01.487072 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.487038    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:53:01.487072 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.487070    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a3a7646-a623-4528-873b-e7dec98465e4-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:53:01.487072 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.487082    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3a7646-a623-4528-873b-e7dec98465e4-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:53:01.487230 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.487090    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a7646-a623-4528-873b-e7dec98465e4-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:53:01.487230 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.487099    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlvgg\" (UniqueName: \"kubernetes.io/projected/8a3a7646-a623-4528-873b-e7dec98465e4-kube-api-access-tlvgg\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:53:01.575117 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.575090    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"]
Sep 17 19:53:01.578887 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:01.578865    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-k8m7g"]
Sep 17 19:53:03.038841 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.038809    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" path="/var/lib/kubelet/pods/8a3a7646-a623-4528-873b-e7dec98465e4/volumes"
Sep 17 19:53:03.556422 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556380    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"]
Sep 17 19:53:03.556729 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556716    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81787dc-8e64-46a0-b26a-7b459074f2e1" containerName="storage-initializer"
Sep 17 19:53:03.556780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556730    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81787dc-8e64-46a0-b26a-7b459074f2e1" containerName="storage-initializer"
Sep 17 19:53:03.556780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556745    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="storage-initializer"
Sep 17 19:53:03.556780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556750    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="storage-initializer"
Sep 17 19:53:03.556780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556759    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81787dc-8e64-46a0-b26a-7b459074f2e1" containerName="storage-initializer"
Sep 17 19:53:03.556780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556765    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81787dc-8e64-46a0-b26a-7b459074f2e1" containerName="storage-initializer"
Sep 17 19:53:03.556780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556776    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container"
Sep 17 19:53:03.556965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556783    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container"
Sep 17 19:53:03.556965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556791    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kube-rbac-proxy"
Sep 17 19:53:03.556965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556796    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kube-rbac-proxy"
Sep 17 19:53:03.556965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556848    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81787dc-8e64-46a0-b26a-7b459074f2e1" containerName="storage-initializer"
Sep 17 19:53:03.556965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556855    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kube-rbac-proxy"
Sep 17 19:53:03.556965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556863    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81787dc-8e64-46a0-b26a-7b459074f2e1" containerName="storage-initializer"
Sep 17 19:53:03.556965 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.556869    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3a7646-a623-4528-873b-e7dec98465e4" containerName="kserve-container"
Sep 17 19:53:03.560118 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.560101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.562383 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.562349    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:53:03.562383 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.562349    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:53:03.562604 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.562437    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 19:53:03.562604 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.562484    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:53:03.562604 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.562558    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 19:53:03.562604 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.562578    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:53:03.563184 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.563167    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:53:03.569325 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.569300    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"]
Sep 17 19:53:03.601096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.601067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.601216 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.601114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nbxt\" (UniqueName: \"kubernetes.io/projected/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-api-access-5nbxt\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.601216 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.601133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.601301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.601220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.601301 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.601255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.701838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.701811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.701987 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.701847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.701987 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.701897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.701987 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:53:03.701971    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 19:53:03.702163 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:53:03.702049    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls podName:fdb3049a-aa0f-4e31-a03a-cf9e7e24d503 nodeName:}" failed. No retries permitted until 2026-09-17 19:53:04.202026233 +0000 UTC m=+2803.329767630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" (UID: "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 19:53:03.702163 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.702068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nbxt\" (UniqueName: \"kubernetes.io/projected/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-api-access-5nbxt\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.702163 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.702110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.702317 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.702294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.702536 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.702518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.702707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.702691    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:03.710490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:03.710470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nbxt\" (UniqueName: \"kubernetes.io/projected/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-api-access-5nbxt\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:04.205579 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:04.205536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:04.206014 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:53:04.205696    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 19:53:04.206014 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:53:04.205770    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls podName:fdb3049a-aa0f-4e31-a03a-cf9e7e24d503 nodeName:}" failed. No retries permitted until 2026-09-17 19:53:05.205754324 +0000 UTC m=+2804.333495703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" (UID: "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 19:53:05.212589 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:05.212554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:05.215198 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:05.215172    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:05.370605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:05.370566    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:05.658139 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:05.658112    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"]
Sep 17 19:53:05.659914 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:53:05.659884    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb3049a_aa0f_4e31_a03a_cf9e7e24d503.slice/crio-c3e5c7e83c854448a0df2838c69b09d8a0ff89b8dfdd7a95337abac7ce3905e2 WatchSource:0}: Error finding container c3e5c7e83c854448a0df2838c69b09d8a0ff89b8dfdd7a95337abac7ce3905e2: Status 404 returned error can't find the container with id c3e5c7e83c854448a0df2838c69b09d8a0ff89b8dfdd7a95337abac7ce3905e2
Sep 17 19:53:06.276154 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:06.276117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" event={"ID":"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503","Type":"ContainerStarted","Data":"a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269"}
Sep 17 19:53:06.276532 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:06.276176    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" event={"ID":"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503","Type":"ContainerStarted","Data":"c3e5c7e83c854448a0df2838c69b09d8a0ff89b8dfdd7a95337abac7ce3905e2"}
Sep 17 19:53:11.298601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:11.298559    2583 generic.go:356] "Generic (PLEG): container finished" podID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerID="a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269" exitCode=0
Sep 17 19:53:11.298986 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:11.298651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" event={"ID":"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503","Type":"ContainerDied","Data":"a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269"}
Sep 17 19:53:12.305281 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:12.305246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" event={"ID":"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503","Type":"ContainerStarted","Data":"b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40"}
Sep 17 19:53:12.305707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:12.305305    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" event={"ID":"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503","Type":"ContainerStarted","Data":"92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8"}
Sep 17 19:53:12.305707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:12.305581    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:12.305798 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:12.305735    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:12.307096 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:12.307069    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 19:53:12.328871 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:12.328828    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" podStartSLOduration=4.714578761 podStartE2EDuration="9.328815329s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.614236568s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:53:12.326212367 +0000 UTC m=+2811.453953772" watchObservedRunningTime="2026-09-17 19:53:12.328815329 +0000 UTC m=+2811.456556730"
Sep 17 19:53:13.309838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:13.309793    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 19:53:18.313911 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:18.313883    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:53:18.314374 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:18.314349    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 19:53:28.315168 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:28.315126    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 19:53:38.314390 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.314350    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 19:53:38.403757 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.403703    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"]
Sep 17 19:53:38.408600 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.408577    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.410891 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.410869    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 19:53:38.411031 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.410945    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 19:53:38.411031 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.410880    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:53:38.416879 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.416855    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"]
Sep 17 19:53:38.464767 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.464734    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88669064-2844-4abb-81d8-9737595ab47a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.464923 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.464785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88669064-2844-4abb-81d8-9737595ab47a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.464923 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.464811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88669064-2844-4abb-81d8-9737595ab47a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.464923 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.464846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvnx\" (UniqueName: \"kubernetes.io/projected/88669064-2844-4abb-81d8-9737595ab47a-kube-api-access-vnvnx\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.565857 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.565750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88669064-2844-4abb-81d8-9737595ab47a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.565857 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.565811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88669064-2844-4abb-81d8-9737595ab47a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.566110 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.565871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvnx\" (UniqueName: \"kubernetes.io/projected/88669064-2844-4abb-81d8-9737595ab47a-kube-api-access-vnvnx\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.566110 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.565916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88669064-2844-4abb-81d8-9737595ab47a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.566247 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.566218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88669064-2844-4abb-81d8-9737595ab47a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.566599 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.566577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88669064-2844-4abb-81d8-9737595ab47a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.568496 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.568479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88669064-2844-4abb-81d8-9737595ab47a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.574129 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.574105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvnx\" (UniqueName: \"kubernetes.io/projected/88669064-2844-4abb-81d8-9737595ab47a-kube-api-access-vnvnx\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.720261 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.720220    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:38.992087 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:38.992060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"]
Sep 17 19:53:38.994108 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:53:38.994080    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88669064_2844_4abb_81d8_9737595ab47a.slice/crio-d873639cf6a7baa6cbdaade4f979a0b1361315a3af2a3f74bb35be3c64cf202d WatchSource:0}: Error finding container d873639cf6a7baa6cbdaade4f979a0b1361315a3af2a3f74bb35be3c64cf202d: Status 404 returned error can't find the container with id d873639cf6a7baa6cbdaade4f979a0b1361315a3af2a3f74bb35be3c64cf202d
Sep 17 19:53:39.418188 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:39.418151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6" event={"ID":"88669064-2844-4abb-81d8-9737595ab47a","Type":"ContainerStarted","Data":"a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4"}
Sep 17 19:53:39.418714 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:39.418219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6" event={"ID":"88669064-2844-4abb-81d8-9737595ab47a","Type":"ContainerStarted","Data":"d873639cf6a7baa6cbdaade4f979a0b1361315a3af2a3f74bb35be3c64cf202d"}
Sep 17 19:53:44.441021 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:44.440993    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6_88669064-2844-4abb-81d8-9737595ab47a/storage-initializer/0.log"
Sep 17 19:53:44.441388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:44.441031    2583 generic.go:356] "Generic (PLEG): container finished" podID="88669064-2844-4abb-81d8-9737595ab47a" containerID="a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4" exitCode=1
Sep 17 19:53:44.441388 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:44.441066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6" event={"ID":"88669064-2844-4abb-81d8-9737595ab47a","Type":"ContainerDied","Data":"a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4"}
Sep 17 19:53:45.446990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:45.446962    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6_88669064-2844-4abb-81d8-9737595ab47a/storage-initializer/0.log"
Sep 17 19:53:45.447347 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:45.447026    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6" event={"ID":"88669064-2844-4abb-81d8-9737595ab47a","Type":"ContainerStarted","Data":"44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e"}
Sep 17 19:53:48.315000 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:48.314947    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 19:53:48.388147 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:48.388106    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"]
Sep 17 19:53:48.388424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:48.388390    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6" podUID="88669064-2844-4abb-81d8-9737595ab47a" containerName="storage-initializer" containerID="cri-o://44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e" gracePeriod=30
Sep 17 19:53:50.035207 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.035186    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6_88669064-2844-4abb-81d8-9737595ab47a/storage-initializer/1.log"
Sep 17 19:53:50.035591 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.035575    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6_88669064-2844-4abb-81d8-9737595ab47a/storage-initializer/0.log"
Sep 17 19:53:50.037909 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.037896    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:50.170404 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.170363    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88669064-2844-4abb-81d8-9737595ab47a-kube-rbac-proxy-sar-config\") pod \"88669064-2844-4abb-81d8-9737595ab47a\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") "
Sep 17 19:53:50.170601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.170467    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88669064-2844-4abb-81d8-9737595ab47a-kserve-provision-location\") pod \"88669064-2844-4abb-81d8-9737595ab47a\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") "
Sep 17 19:53:50.170601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.170491    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88669064-2844-4abb-81d8-9737595ab47a-proxy-tls\") pod \"88669064-2844-4abb-81d8-9737595ab47a\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") "
Sep 17 19:53:50.170601 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.170533    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnvnx\" (UniqueName: \"kubernetes.io/projected/88669064-2844-4abb-81d8-9737595ab47a-kube-api-access-vnvnx\") pod \"88669064-2844-4abb-81d8-9737595ab47a\" (UID: \"88669064-2844-4abb-81d8-9737595ab47a\") "
Sep 17 19:53:50.170802 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.170770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88669064-2844-4abb-81d8-9737595ab47a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88669064-2844-4abb-81d8-9737595ab47a" (UID: "88669064-2844-4abb-81d8-9737595ab47a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:53:50.170859 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.170795    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88669064-2844-4abb-81d8-9737595ab47a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88669064-2844-4abb-81d8-9737595ab47a" (UID: "88669064-2844-4abb-81d8-9737595ab47a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:53:50.171031 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.170998    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88669064-2844-4abb-81d8-9737595ab47a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:53:50.171031 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.171034    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88669064-2844-4abb-81d8-9737595ab47a-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:53:50.172884 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.172864    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88669064-2844-4abb-81d8-9737595ab47a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88669064-2844-4abb-81d8-9737595ab47a" (UID: "88669064-2844-4abb-81d8-9737595ab47a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:53:50.172988 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.172930    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88669064-2844-4abb-81d8-9737595ab47a-kube-api-access-vnvnx" (OuterVolumeSpecName: "kube-api-access-vnvnx") pod "88669064-2844-4abb-81d8-9737595ab47a" (UID: "88669064-2844-4abb-81d8-9737595ab47a"). InnerVolumeSpecName "kube-api-access-vnvnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:53:50.271424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.271391    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88669064-2844-4abb-81d8-9737595ab47a-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:53:50.271424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.271421    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnvnx\" (UniqueName: \"kubernetes.io/projected/88669064-2844-4abb-81d8-9737595ab47a-kube-api-access-vnvnx\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:53:50.470994 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.470917    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6_88669064-2844-4abb-81d8-9737595ab47a/storage-initializer/1.log"
Sep 17 19:53:50.471341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.471326    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6_88669064-2844-4abb-81d8-9737595ab47a/storage-initializer/0.log"
Sep 17 19:53:50.471398 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.471365    2583 generic.go:356] "Generic (PLEG): container finished" podID="88669064-2844-4abb-81d8-9737595ab47a" containerID="44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e" exitCode=1
Sep 17 19:53:50.471449 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.471433    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"
Sep 17 19:53:50.471491 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.471452    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6" event={"ID":"88669064-2844-4abb-81d8-9737595ab47a","Type":"ContainerDied","Data":"44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e"}
Sep 17 19:53:50.471528 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.471514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6" event={"ID":"88669064-2844-4abb-81d8-9737595ab47a","Type":"ContainerDied","Data":"d873639cf6a7baa6cbdaade4f979a0b1361315a3af2a3f74bb35be3c64cf202d"}
Sep 17 19:53:50.471565 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.471541    2583 scope.go:117] "RemoveContainer" containerID="44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e"
Sep 17 19:53:50.480352 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.480336    2583 scope.go:117] "RemoveContainer" containerID="a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4"
Sep 17 19:53:50.489472 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.489451    2583 scope.go:117] "RemoveContainer" containerID="44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e"
Sep 17 19:53:50.489813 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:53:50.489790    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e\": container with ID starting with 44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e not found: ID does not exist" containerID="44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e"
Sep 17 19:53:50.489908 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.489826    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e"} err="failed to get container status \"44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e\": rpc error: code = NotFound desc = could not find container \"44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e\": container with ID starting with 44512c471a3f172c3bb3ca6ddb6eb53db89dfc198c458e5759d4a1d0c5c5b36e not found: ID does not exist"
Sep 17 19:53:50.489908 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.489850    2583 scope.go:117] "RemoveContainer" containerID="a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4"
Sep 17 19:53:50.490119 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:53:50.490103    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4\": container with ID starting with a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4 not found: ID does not exist" containerID="a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4"
Sep 17 19:53:50.490157 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.490125    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4"} err="failed to get container status \"a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4\": rpc error: code = NotFound desc = could not find container \"a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4\": container with ID starting with a670800979692e0811f747bbbe17e85f10608ef7c447f55a6a5e80295c38d2a4 not found: ID does not exist"
Sep 17 19:53:50.513182 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.513154    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"]
Sep 17 19:53:50.519275 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:50.519251    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-nqxl6"]
Sep 17 19:53:51.038742 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:51.038713    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88669064-2844-4abb-81d8-9737595ab47a" path="/var/lib/kubelet/pods/88669064-2844-4abb-81d8-9737595ab47a/volumes"
Sep 17 19:53:58.315323 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:53:58.315292    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:54:05.178595 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:05.178513    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"]
Sep 17 19:54:05.178988 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:05.178806    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container" containerID="cri-o://92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8" gracePeriod=30
Sep 17 19:54:05.178988 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:05.178869    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kube-rbac-proxy" containerID="cri-o://b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40" gracePeriod=30
Sep 17 19:54:05.542092 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:05.542006    2583 generic.go:356] "Generic (PLEG): container finished" podID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerID="b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40" exitCode=2
Sep 17 19:54:05.542223 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:05.542083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" event={"ID":"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503","Type":"ContainerDied","Data":"b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40"}
Sep 17 19:54:08.126585 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.126561    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:54:08.218143 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.218061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-rbac-proxy-sar-config\") pod \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") "
Sep 17 19:54:08.218143 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.218104    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nbxt\" (UniqueName: \"kubernetes.io/projected/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-api-access-5nbxt\") pod \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") "
Sep 17 19:54:08.218143 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.218124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-cabundle-cert\") pod \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") "
Sep 17 19:54:08.218368 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.218193    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls\") pod \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") "
Sep 17 19:54:08.218368 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.218248    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kserve-provision-location\") pod \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\" (UID: \"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503\") "
Sep 17 19:54:08.218514 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.218485    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" (UID: "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:54:08.218653 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.218544    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" (UID: "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:54:08.220460 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.220434    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-api-access-5nbxt" (OuterVolumeSpecName: "kube-api-access-5nbxt") pod "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" (UID: "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503"). InnerVolumeSpecName "kube-api-access-5nbxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:54:08.220525 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.220452    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" (UID: "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:54:08.228255 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.228232    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" (UID: "fdb3049a-aa0f-4e31-a03a-cf9e7e24d503"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:54:08.319523 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.319489    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:54:08.319523 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.319516    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:54:08.319523 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.319526    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nbxt\" (UniqueName: \"kubernetes.io/projected/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-kube-api-access-5nbxt\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:54:08.319790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.319535    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:54:08.319790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.319545    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:54:08.556968 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.556873    2583 generic.go:356] "Generic (PLEG): container finished" podID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerID="92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8" exitCode=0
Sep 17 19:54:08.556968 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.556959    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" event={"ID":"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503","Type":"ContainerDied","Data":"92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8"}
Sep 17 19:54:08.557150 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.556935    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"
Sep 17 19:54:08.557150 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.557046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759" event={"ID":"fdb3049a-aa0f-4e31-a03a-cf9e7e24d503","Type":"ContainerDied","Data":"c3e5c7e83c854448a0df2838c69b09d8a0ff89b8dfdd7a95337abac7ce3905e2"}
Sep 17 19:54:08.557150 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.557074    2583 scope.go:117] "RemoveContainer" containerID="b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40"
Sep 17 19:54:08.566009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.565993    2583 scope.go:117] "RemoveContainer" containerID="92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8"
Sep 17 19:54:08.573813 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.573793    2583 scope.go:117] "RemoveContainer" containerID="a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269"
Sep 17 19:54:08.581221 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.581204    2583 scope.go:117] "RemoveContainer" containerID="b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40"
Sep 17 19:54:08.581668 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:54:08.581575    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40\": container with ID starting with b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40 not found: ID does not exist" containerID="b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40"
Sep 17 19:54:08.581668 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.581633    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40"} err="failed to get container status \"b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40\": rpc error: code = NotFound desc = could not find container \"b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40\": container with ID starting with b3c65d26a20cee545deb85c13acb2975208e81c1dc12685b19df323d989a0d40 not found: ID does not exist"
Sep 17 19:54:08.581668 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.581660    2583 scope.go:117] "RemoveContainer" containerID="92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8"
Sep 17 19:54:08.582030 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:54:08.582002    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8\": container with ID starting with 92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8 not found: ID does not exist" containerID="92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8"
Sep 17 19:54:08.582164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.582036    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8"} err="failed to get container status \"92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8\": rpc error: code = NotFound desc = could not find container \"92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8\": container with ID starting with 92dcb129174ec9d843b879d061b6ba04ac53c0dc19a4c1b21e8a104e128f3be8 not found: ID does not exist"
Sep 17 19:54:08.582164 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.582068    2583 scope.go:117] "RemoveContainer" containerID="a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269"
Sep 17 19:54:08.582401 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:54:08.582360    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269\": container with ID starting with a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269 not found: ID does not exist" containerID="a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269"
Sep 17 19:54:08.582401 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.582391    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269"} err="failed to get container status \"a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269\": rpc error: code = NotFound desc = could not find container \"a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269\": container with ID starting with a1fc6e06a84bb00ab351165cac1079b569381ce38437baabf362cbe96f7fb269 not found: ID does not exist"
Sep 17 19:54:08.583881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.583863    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"]
Sep 17 19:54:08.589950 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:08.589930    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-4z759"]
Sep 17 19:54:09.037821 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:09.037789    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" path="/var/lib/kubelet/pods/fdb3049a-aa0f-4e31-a03a-cf9e7e24d503/volumes"
Sep 17 19:54:10.486341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486312    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"]
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486629    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="storage-initializer"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486641    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="storage-initializer"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486651    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88669064-2844-4abb-81d8-9737595ab47a" containerName="storage-initializer"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486656    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="88669064-2844-4abb-81d8-9737595ab47a" containerName="storage-initializer"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486665    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88669064-2844-4abb-81d8-9737595ab47a" containerName="storage-initializer"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486670    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="88669064-2844-4abb-81d8-9737595ab47a" containerName="storage-initializer"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486679    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486684    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486706    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kube-rbac-proxy"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486711    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kube-rbac-proxy"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486765    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="88669064-2844-4abb-81d8-9737595ab47a" containerName="storage-initializer"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486773    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kserve-container"
Sep 17 19:54:10.486838 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486779    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdb3049a-aa0f-4e31-a03a-cf9e7e24d503" containerName="kube-rbac-proxy"
Sep 17 19:54:10.487235 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.486863    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="88669064-2844-4abb-81d8-9737595ab47a" containerName="storage-initializer"
Sep 17 19:54:10.491454 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.491436    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.494140 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.494120    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 19:54:10.494377 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.494360    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:54:10.494698 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.494677    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 19:54:10.494790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.494704    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:54:10.494790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.494716    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:54:10.494790 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.494680    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:54:10.495019 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.495003    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:54:10.501636 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.501597    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"]
Sep 17 19:54:10.536990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.536962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.536990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.536991    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bt2n\" (UniqueName: \"kubernetes.io/projected/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-api-access-6bt2n\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.537137 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.537028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef92072a-cafb-4dee-9c72-00f3bb624272-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.537137 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.537112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef92072a-cafb-4dee-9c72-00f3bb624272-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.537211 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.537156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.638112 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.638080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef92072a-cafb-4dee-9c72-00f3bb624272-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.638251 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.638126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef92072a-cafb-4dee-9c72-00f3bb624272-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.638251 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.638169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.638251 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.638214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.638251 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.638239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bt2n\" (UniqueName: \"kubernetes.io/projected/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-api-access-6bt2n\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.638556 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.638538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef92072a-cafb-4dee-9c72-00f3bb624272-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.638841 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.638820    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.638920 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.638884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.640760 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.640741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef92072a-cafb-4dee-9c72-00f3bb624272-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.647825 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.647804    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bt2n\" (UniqueName: \"kubernetes.io/projected/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-api-access-6bt2n\") pod \"isvc-pmml-predictor-8674795f59-b8flp\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:10.802633 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:10.802524    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:11.081725 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:11.081697    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"]
Sep 17 19:54:11.084086 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:54:11.084060    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef92072a_cafb_4dee_9c72_00f3bb624272.slice/crio-2d2fa4e429d921ddb2b9bf274e8f3aefe6f6ac17df2f2e291120dfaea2ec081e WatchSource:0}: Error finding container 2d2fa4e429d921ddb2b9bf274e8f3aefe6f6ac17df2f2e291120dfaea2ec081e: Status 404 returned error can't find the container with id 2d2fa4e429d921ddb2b9bf274e8f3aefe6f6ac17df2f2e291120dfaea2ec081e
Sep 17 19:54:11.571782 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:11.571743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" event={"ID":"ef92072a-cafb-4dee-9c72-00f3bb624272","Type":"ContainerStarted","Data":"4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1"}
Sep 17 19:54:11.572165 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:11.571814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" event={"ID":"ef92072a-cafb-4dee-9c72-00f3bb624272","Type":"ContainerStarted","Data":"2d2fa4e429d921ddb2b9bf274e8f3aefe6f6ac17df2f2e291120dfaea2ec081e"}
Sep 17 19:54:15.590007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:15.589927    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerID="4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1" exitCode=0
Sep 17 19:54:15.590341 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:15.589999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" event={"ID":"ef92072a-cafb-4dee-9c72-00f3bb624272","Type":"ContainerDied","Data":"4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1"}
Sep 17 19:54:23.633377 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:23.633343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" event={"ID":"ef92072a-cafb-4dee-9c72-00f3bb624272","Type":"ContainerStarted","Data":"e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a"}
Sep 17 19:54:23.633758 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:23.633404    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" event={"ID":"ef92072a-cafb-4dee-9c72-00f3bb624272","Type":"ContainerStarted","Data":"0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a"}
Sep 17 19:54:23.633758 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:23.633673    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:23.633839 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:23.633807    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:23.634990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:23.634965    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:54:23.653153 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:23.653110    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podStartSLOduration=2.38157833 podStartE2EDuration="13.653098594s" totalImagesPullingTime="7.244641607s" totalInitContainerRuntime="4.026878657s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:54:23.651191335 +0000 UTC m=+2882.778932737" watchObservedRunningTime="2026-09-17 19:54:23.653098594 +0000 UTC m=+2882.780839994"
Sep 17 19:54:24.638278 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:24.638241    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:54:29.642714 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:29.642684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:54:29.643171 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:29.643145    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:54:39.644116 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:39.644071    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:54:49.643907 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:49.643865    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:54:59.643870 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:54:59.643830    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:55:09.643890 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:09.643848    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:55:19.643761 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:19.643715    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:55:20.662788 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.662752    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"]
Sep 17 19:55:20.691653 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.691605    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"]
Sep 17 19:55:20.691806 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.691707    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.694101 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.694074    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 19:55:20.694254 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.694145    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:55:20.694746 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.694729    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 19:55:20.721331 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.721306    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a8146d-5315-4095-893b-b8c018834041-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.721462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.721347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz7j8\" (UniqueName: \"kubernetes.io/projected/17a8146d-5315-4095-893b-b8c018834041-kube-api-access-dz7j8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.721462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.721395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a8146d-5315-4095-893b-b8c018834041-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.721462 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.721428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a8146d-5315-4095-893b-b8c018834041-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.822242 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.822212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a8146d-5315-4095-893b-b8c018834041-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.822424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.822265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a8146d-5315-4095-893b-b8c018834041-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.822424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.822295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz7j8\" (UniqueName: \"kubernetes.io/projected/17a8146d-5315-4095-893b-b8c018834041-kube-api-access-dz7j8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.822424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.822336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a8146d-5315-4095-893b-b8c018834041-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.822424 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:55:20.822353    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 19:55:20.822424 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:55:20.822423    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a8146d-5315-4095-893b-b8c018834041-proxy-tls podName:17a8146d-5315-4095-893b-b8c018834041 nodeName:}" failed. No retries permitted until 2026-09-17 19:55:21.322406122 +0000 UTC m=+2940.450147501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/17a8146d-5315-4095-893b-b8c018834041-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln" (UID: "17a8146d-5315-4095-893b-b8c018834041") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 19:55:20.822886 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.822865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a8146d-5315-4095-893b-b8c018834041-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.823054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.823034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a8146d-5315-4095-893b-b8c018834041-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:20.831143 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:20.831120    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz7j8\" (UniqueName: \"kubernetes.io/projected/17a8146d-5315-4095-893b-b8c018834041-kube-api-access-dz7j8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:21.326537 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:21.326505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a8146d-5315-4095-893b-b8c018834041-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:21.329065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:21.329045    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a8146d-5315-4095-893b-b8c018834041-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:21.601384 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:21.601304    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:21.875442 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:21.875417    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"]
Sep 17 19:55:21.877724 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:55:21.877697    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a8146d_5315_4095_893b_b8c018834041.slice/crio-5780b17155b68e48634dfba20943bef64ac1479114d79485fa2c7990f6aa302a WatchSource:0}: Error finding container 5780b17155b68e48634dfba20943bef64ac1479114d79485fa2c7990f6aa302a: Status 404 returned error can't find the container with id 5780b17155b68e48634dfba20943bef64ac1479114d79485fa2c7990f6aa302a
Sep 17 19:55:22.877362 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:22.877328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln" event={"ID":"17a8146d-5315-4095-893b-b8c018834041","Type":"ContainerStarted","Data":"5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d"}
Sep 17 19:55:22.877765 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:22.877385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln" event={"ID":"17a8146d-5315-4095-893b-b8c018834041","Type":"ContainerStarted","Data":"5780b17155b68e48634dfba20943bef64ac1479114d79485fa2c7990f6aa302a"}
Sep 17 19:55:25.894043 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:25.894011    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln_17a8146d-5315-4095-893b-b8c018834041/storage-initializer/0.log"
Sep 17 19:55:25.894496 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:25.894054    2583 generic.go:356] "Generic (PLEG): container finished" podID="17a8146d-5315-4095-893b-b8c018834041" containerID="5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d" exitCode=1
Sep 17 19:55:25.894496 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:25.894116    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln" event={"ID":"17a8146d-5315-4095-893b-b8c018834041","Type":"ContainerDied","Data":"5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d"}
Sep 17 19:55:26.900498 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:26.900471    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln_17a8146d-5315-4095-893b-b8c018834041/storage-initializer/0.log"
Sep 17 19:55:26.900899 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:26.900575    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln" event={"ID":"17a8146d-5315-4095-893b-b8c018834041","Type":"ContainerStarted","Data":"b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa"}
Sep 17 19:55:29.643439 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:29.643399    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:55:30.622258 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:30.622218    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"]
Sep 17 19:55:30.622490 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:30.622467    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln" podUID="17a8146d-5315-4095-893b-b8c018834041" containerName="storage-initializer" containerID="cri-o://b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa" gracePeriod=30
Sep 17 19:55:32.557260 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.557239    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln_17a8146d-5315-4095-893b-b8c018834041/storage-initializer/1.log"
Sep 17 19:55:32.557651 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.557636    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln_17a8146d-5315-4095-893b-b8c018834041/storage-initializer/0.log"
Sep 17 19:55:32.558964 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.558948    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:32.627261 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.627232    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a8146d-5315-4095-893b-b8c018834041-kube-rbac-proxy-sar-config\") pod \"17a8146d-5315-4095-893b-b8c018834041\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") "
Sep 17 19:55:32.627417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.627272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a8146d-5315-4095-893b-b8c018834041-proxy-tls\") pod \"17a8146d-5315-4095-893b-b8c018834041\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") "
Sep 17 19:55:32.627417 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.627307    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a8146d-5315-4095-893b-b8c018834041-kserve-provision-location\") pod \"17a8146d-5315-4095-893b-b8c018834041\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") "
Sep 17 19:55:32.627503 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.627416    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz7j8\" (UniqueName: \"kubernetes.io/projected/17a8146d-5315-4095-893b-b8c018834041-kube-api-access-dz7j8\") pod \"17a8146d-5315-4095-893b-b8c018834041\" (UID: \"17a8146d-5315-4095-893b-b8c018834041\") "
Sep 17 19:55:32.627541 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.627512    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a8146d-5315-4095-893b-b8c018834041-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17a8146d-5315-4095-893b-b8c018834041" (UID: "17a8146d-5315-4095-893b-b8c018834041"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:32.627659 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.627636    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a8146d-5315-4095-893b-b8c018834041-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "17a8146d-5315-4095-893b-b8c018834041" (UID: "17a8146d-5315-4095-893b-b8c018834041"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:55:32.627707 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.627664    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a8146d-5315-4095-893b-b8c018834041-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:55:32.629371 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.629347    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a8146d-5315-4095-893b-b8c018834041-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17a8146d-5315-4095-893b-b8c018834041" (UID: "17a8146d-5315-4095-893b-b8c018834041"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:55:32.629484 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.629466    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a8146d-5315-4095-893b-b8c018834041-kube-api-access-dz7j8" (OuterVolumeSpecName: "kube-api-access-dz7j8") pod "17a8146d-5315-4095-893b-b8c018834041" (UID: "17a8146d-5315-4095-893b-b8c018834041"). InnerVolumeSpecName "kube-api-access-dz7j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:55:32.728242 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.728218    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a8146d-5315-4095-893b-b8c018834041-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:55:32.728242 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.728240    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a8146d-5315-4095-893b-b8c018834041-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:55:32.728406 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.728250    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz7j8\" (UniqueName: \"kubernetes.io/projected/17a8146d-5315-4095-893b-b8c018834041-kube-api-access-dz7j8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:55:32.929453 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.929428    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln_17a8146d-5315-4095-893b-b8c018834041/storage-initializer/1.log"
Sep 17 19:55:32.929894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.929877    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln_17a8146d-5315-4095-893b-b8c018834041/storage-initializer/0.log"
Sep 17 19:55:32.929967 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.929913    2583 generic.go:356] "Generic (PLEG): container finished" podID="17a8146d-5315-4095-893b-b8c018834041" containerID="b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa" exitCode=1
Sep 17 19:55:32.930012 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.929972    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"
Sep 17 19:55:32.930012 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.930001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln" event={"ID":"17a8146d-5315-4095-893b-b8c018834041","Type":"ContainerDied","Data":"b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa"}
Sep 17 19:55:32.930095 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.930055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln" event={"ID":"17a8146d-5315-4095-893b-b8c018834041","Type":"ContainerDied","Data":"5780b17155b68e48634dfba20943bef64ac1479114d79485fa2c7990f6aa302a"}
Sep 17 19:55:32.930095 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.930081    2583 scope.go:117] "RemoveContainer" containerID="b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa"
Sep 17 19:55:32.939009 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.938990    2583 scope.go:117] "RemoveContainer" containerID="5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d"
Sep 17 19:55:32.946910 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.946893    2583 scope.go:117] "RemoveContainer" containerID="b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa"
Sep 17 19:55:32.947158 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:55:32.947139    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa\": container with ID starting with b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa not found: ID does not exist" containerID="b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa"
Sep 17 19:55:32.947209 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.947167    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa"} err="failed to get container status \"b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa\": rpc error: code = NotFound desc = could not find container \"b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa\": container with ID starting with b1c4f810cc6cb9fadd4d58f1ae4595c62f970ac1a8b29801cae2b4f991de46aa not found: ID does not exist"
Sep 17 19:55:32.947209 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.947186    2583 scope.go:117] "RemoveContainer" containerID="5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d"
Sep 17 19:55:32.947467 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:55:32.947449    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d\": container with ID starting with 5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d not found: ID does not exist" containerID="5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d"
Sep 17 19:55:32.947524 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.947472    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d"} err="failed to get container status \"5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d\": rpc error: code = NotFound desc = could not find container \"5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d\": container with ID starting with 5968724839e46140802e5a98887624935912193cdbea1a2246d5d43aac2e684d not found: ID does not exist"
Sep 17 19:55:32.969226 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.969196    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"]
Sep 17 19:55:32.971567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:32.971541    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-5pxln"]
Sep 17 19:55:33.038708 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:33.038639    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a8146d-5315-4095-893b-b8c018834041" path="/var/lib/kubelet/pods/17a8146d-5315-4095-893b-b8c018834041/volumes"
Sep 17 19:55:35.034430 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:35.034390    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:55:45.034813 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:45.034767    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:55:55.038443 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:55:55.038415    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:56:02.087722 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:02.087681    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"]
Sep 17 19:56:02.088216 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:02.088003    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" containerID="cri-o://0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a" gracePeriod=30
Sep 17 19:56:02.088216 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:02.088047    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kube-rbac-proxy" containerID="cri-o://e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a" gracePeriod=30
Sep 17 19:56:03.061548 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:03.061516    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerID="e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a" exitCode=2
Sep 17 19:56:03.061548 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:03.061554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" event={"ID":"ef92072a-cafb-4dee-9c72-00f3bb624272","Type":"ContainerDied","Data":"e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a"}
Sep 17 19:56:04.639004 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:04.638962    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.71:8643/healthz\": dial tcp 10.134.0.71:8643: connect: connection refused"
Sep 17 19:56:05.035007 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:05.034964    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 17 19:56:06.233263 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.233237    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:56:06.294952 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.294884    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef92072a-cafb-4dee-9c72-00f3bb624272-proxy-tls\") pod \"ef92072a-cafb-4dee-9c72-00f3bb624272\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") "
Sep 17 19:56:06.294952 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.294948    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-rbac-proxy-sar-config\") pod \"ef92072a-cafb-4dee-9c72-00f3bb624272\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") "
Sep 17 19:56:06.295145 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.294967    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bt2n\" (UniqueName: \"kubernetes.io/projected/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-api-access-6bt2n\") pod \"ef92072a-cafb-4dee-9c72-00f3bb624272\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") "
Sep 17 19:56:06.295145 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.294990    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-cabundle-cert\") pod \"ef92072a-cafb-4dee-9c72-00f3bb624272\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") "
Sep 17 19:56:06.295145 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.295013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef92072a-cafb-4dee-9c72-00f3bb624272-kserve-provision-location\") pod \"ef92072a-cafb-4dee-9c72-00f3bb624272\" (UID: \"ef92072a-cafb-4dee-9c72-00f3bb624272\") "
Sep 17 19:56:06.295389 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.295367    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef92072a-cafb-4dee-9c72-00f3bb624272" (UID: "ef92072a-cafb-4dee-9c72-00f3bb624272"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:56:06.295463 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.295382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef92072a-cafb-4dee-9c72-00f3bb624272" (UID: "ef92072a-cafb-4dee-9c72-00f3bb624272"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:56:06.295463 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.295389    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef92072a-cafb-4dee-9c72-00f3bb624272-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef92072a-cafb-4dee-9c72-00f3bb624272" (UID: "ef92072a-cafb-4dee-9c72-00f3bb624272"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:56:06.297263 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.297238    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-api-access-6bt2n" (OuterVolumeSpecName: "kube-api-access-6bt2n") pod "ef92072a-cafb-4dee-9c72-00f3bb624272" (UID: "ef92072a-cafb-4dee-9c72-00f3bb624272"). InnerVolumeSpecName "kube-api-access-6bt2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:56:06.297263 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.297245    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef92072a-cafb-4dee-9c72-00f3bb624272-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef92072a-cafb-4dee-9c72-00f3bb624272" (UID: "ef92072a-cafb-4dee-9c72-00f3bb624272"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:56:06.395823 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.395783    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef92072a-cafb-4dee-9c72-00f3bb624272-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:56:06.395823 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.395813    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:56:06.395823 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.395823    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bt2n\" (UniqueName: \"kubernetes.io/projected/ef92072a-cafb-4dee-9c72-00f3bb624272-kube-api-access-6bt2n\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:56:06.395823 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.395832    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef92072a-cafb-4dee-9c72-00f3bb624272-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:56:06.396100 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:06.395841    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef92072a-cafb-4dee-9c72-00f3bb624272-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:56:07.079896 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.079866    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerID="0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a" exitCode=0
Sep 17 19:56:07.080054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.079933    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"
Sep 17 19:56:07.080054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.079944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" event={"ID":"ef92072a-cafb-4dee-9c72-00f3bb624272","Type":"ContainerDied","Data":"0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a"}
Sep 17 19:56:07.080054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.080005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp" event={"ID":"ef92072a-cafb-4dee-9c72-00f3bb624272","Type":"ContainerDied","Data":"2d2fa4e429d921ddb2b9bf274e8f3aefe6f6ac17df2f2e291120dfaea2ec081e"}
Sep 17 19:56:07.080054 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.080032    2583 scope.go:117] "RemoveContainer" containerID="e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a"
Sep 17 19:56:07.088919 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.088902    2583 scope.go:117] "RemoveContainer" containerID="0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a"
Sep 17 19:56:07.096763 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.096736    2583 scope.go:117] "RemoveContainer" containerID="4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1"
Sep 17 19:56:07.099567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.099538    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"]
Sep 17 19:56:07.103029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.103005    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b8flp"]
Sep 17 19:56:07.106316 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.106089    2583 scope.go:117] "RemoveContainer" containerID="e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a"
Sep 17 19:56:07.106396 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:56:07.106353    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a\": container with ID starting with e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a not found: ID does not exist" containerID="e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a"
Sep 17 19:56:07.106396 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.106379    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a"} err="failed to get container status \"e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a\": rpc error: code = NotFound desc = could not find container \"e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a\": container with ID starting with e00e91120a50033d6c4ff5c8b54acd05c9cba125d9ccdc0480d2d8715e39123a not found: ID does not exist"
Sep 17 19:56:07.106396 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.106396    2583 scope.go:117] "RemoveContainer" containerID="0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a"
Sep 17 19:56:07.106699 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:56:07.106678    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a\": container with ID starting with 0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a not found: ID does not exist" containerID="0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a"
Sep 17 19:56:07.106777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.106705    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a"} err="failed to get container status \"0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a\": rpc error: code = NotFound desc = could not find container \"0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a\": container with ID starting with 0a94d3f9f18e03e1532857cdefbb555a8e6a7d2b3a7d52213b3f2bd28a0fa96a not found: ID does not exist"
Sep 17 19:56:07.106777 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.106722    2583 scope.go:117] "RemoveContainer" containerID="4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1"
Sep 17 19:56:07.106966 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:56:07.106944    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1\": container with ID starting with 4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1 not found: ID does not exist" containerID="4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1"
Sep 17 19:56:07.107025 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.106972    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1"} err="failed to get container status \"4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1\": rpc error: code = NotFound desc = could not find container \"4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1\": container with ID starting with 4deacb9cc2d7259eb75130a12787e0a2a454ef7bb68d452a20d80c9dde1c59d1 not found: ID does not exist"
Sep 17 19:56:07.266261 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266228    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"]
Sep 17 19:56:07.266605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266535    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a8146d-5315-4095-893b-b8c018834041" containerName="storage-initializer"
Sep 17 19:56:07.266605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266546    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a8146d-5315-4095-893b-b8c018834041" containerName="storage-initializer"
Sep 17 19:56:07.266605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266556    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a8146d-5315-4095-893b-b8c018834041" containerName="storage-initializer"
Sep 17 19:56:07.266605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266562    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a8146d-5315-4095-893b-b8c018834041" containerName="storage-initializer"
Sep 17 19:56:07.266605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266569    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="storage-initializer"
Sep 17 19:56:07.266605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266575    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="storage-initializer"
Sep 17 19:56:07.266605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266594    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kube-rbac-proxy"
Sep 17 19:56:07.266605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266599    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kube-rbac-proxy"
Sep 17 19:56:07.266605 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266635    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container"
Sep 17 19:56:07.266939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266641    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container"
Sep 17 19:56:07.266939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266689    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a8146d-5315-4095-893b-b8c018834041" containerName="storage-initializer"
Sep 17 19:56:07.266939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266696    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kserve-container"
Sep 17 19:56:07.266939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266702    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" containerName="kube-rbac-proxy"
Sep 17 19:56:07.266939 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.266789    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a8146d-5315-4095-893b-b8c018834041" containerName="storage-initializer"
Sep 17 19:56:07.270986 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.270967    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.274029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.273583    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 19:56:07.274029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.273647    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:56:07.274029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.273659    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:56:07.274029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.273687    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:56:07.274029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.273708    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:56:07.274029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.273926    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:56:07.274029 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.274011    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:56:07.280351 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.280331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"]
Sep 17 19:56:07.302138 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.302066    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c3d388-0def-4829-b231-2c8db4becaa2-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.302138 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.302125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.302328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.302157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpmv\" (UniqueName: \"kubernetes.io/projected/86c3d388-0def-4829-b231-2c8db4becaa2-kube-api-access-4fpmv\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.302328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.302259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.302328 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.302293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c3d388-0def-4829-b231-2c8db4becaa2-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.403348 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.403295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c3d388-0def-4829-b231-2c8db4becaa2-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.403348 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.403354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.403599 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.403387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpmv\" (UniqueName: \"kubernetes.io/projected/86c3d388-0def-4829-b231-2c8db4becaa2-kube-api-access-4fpmv\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.403599 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.403426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.403599 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.403441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c3d388-0def-4829-b231-2c8db4becaa2-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.403599 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:56:07.403462    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 17 19:56:07.403599 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:56:07.403553    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86c3d388-0def-4829-b231-2c8db4becaa2-proxy-tls podName:86c3d388-0def-4829-b231-2c8db4becaa2 nodeName:}" failed. No retries permitted until 2026-09-17 19:56:07.903531346 +0000 UTC m=+2987.031272745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/86c3d388-0def-4829-b231-2c8db4becaa2-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-td4bh" (UID: "86c3d388-0def-4829-b231-2c8db4becaa2") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 17 19:56:07.403922 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.403902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c3d388-0def-4829-b231-2c8db4becaa2-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.404135 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.404118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.404168 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.404149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.411894 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.411875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpmv\" (UniqueName: \"kubernetes.io/projected/86c3d388-0def-4829-b231-2c8db4becaa2-kube-api-access-4fpmv\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.907334 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.907300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c3d388-0def-4829-b231-2c8db4becaa2-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:07.910039 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:07.910008    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c3d388-0def-4829-b231-2c8db4becaa2-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-td4bh\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:08.181627 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:08.181585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:08.451780 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:08.451667    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"]
Sep 17 19:56:08.454059 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:56:08.454034    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c3d388_0def_4829_b231_2c8db4becaa2.slice/crio-d90d4827853cdd6246630e70b5e17f03e9fd392701cd366424699de247be8955 WatchSource:0}: Error finding container d90d4827853cdd6246630e70b5e17f03e9fd392701cd366424699de247be8955: Status 404 returned error can't find the container with id d90d4827853cdd6246630e70b5e17f03e9fd392701cd366424699de247be8955
Sep 17 19:56:09.041484 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:09.041440    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef92072a-cafb-4dee-9c72-00f3bb624272" path="/var/lib/kubelet/pods/ef92072a-cafb-4dee-9c72-00f3bb624272/volumes"
Sep 17 19:56:09.091153 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:09.091115    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" event={"ID":"86c3d388-0def-4829-b231-2c8db4becaa2","Type":"ContainerStarted","Data":"bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4"}
Sep 17 19:56:09.091318 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:09.091190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" event={"ID":"86c3d388-0def-4829-b231-2c8db4becaa2","Type":"ContainerStarted","Data":"d90d4827853cdd6246630e70b5e17f03e9fd392701cd366424699de247be8955"}
Sep 17 19:56:13.108238 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:13.108202    2583 generic.go:356] "Generic (PLEG): container finished" podID="86c3d388-0def-4829-b231-2c8db4becaa2" containerID="bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4" exitCode=0
Sep 17 19:56:13.108676 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:13.108273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" event={"ID":"86c3d388-0def-4829-b231-2c8db4becaa2","Type":"ContainerDied","Data":"bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4"}
Sep 17 19:56:14.114908 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:14.114876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" event={"ID":"86c3d388-0def-4829-b231-2c8db4becaa2","Type":"ContainerStarted","Data":"221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f"}
Sep 17 19:56:14.115445 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:14.114934    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" event={"ID":"86c3d388-0def-4829-b231-2c8db4becaa2","Type":"ContainerStarted","Data":"e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0"}
Sep 17 19:56:14.115445 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:14.115245    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:14.115445 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:14.115386    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:14.116595 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:14.116575    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:56:14.133294 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:14.133254    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podStartSLOduration=3.54197718 podStartE2EDuration="7.133238638s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.591261458s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:56:14.132903327 +0000 UTC m=+2993.260644729" watchObservedRunningTime="2026-09-17 19:56:14.133238638 +0000 UTC m=+2993.260980043"
Sep 17 19:56:15.119175 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:15.119135    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:56:16.123418 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:16.123380    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:56:21.128457 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:21.128425    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:56:21.128928 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:21.128901    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:56:27.912261 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:27.912231    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:56:27.918800 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:27.918778    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 19:56:31.129113 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:31.129077    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:56:41.129610 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:41.129570    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:56:51.129579 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:56:51.129537    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:57:01.129022 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:01.128982    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:57:11.129520 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:11.129437    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:57:21.129851 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:21.129805    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:57:31.129228 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:31.129188    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:57:41.130259 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:41.130224    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:57:48.987498 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:48.987465    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"]
Sep 17 19:57:48.987926 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:48.987730    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" containerID="cri-o://e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0" gracePeriod=30
Sep 17 19:57:48.987926 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:48.987775    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kube-rbac-proxy" containerID="cri-o://221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f" gracePeriod=30
Sep 17 19:57:49.509447 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:49.509417    2583 generic.go:356] "Generic (PLEG): container finished" podID="86c3d388-0def-4829-b231-2c8db4becaa2" containerID="221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f" exitCode=2
Sep 17 19:57:49.509644 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:49.509499    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" event={"ID":"86c3d388-0def-4829-b231-2c8db4becaa2","Type":"ContainerDied","Data":"221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f"}
Sep 17 19:57:51.123792 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:51.123747    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.73:8643/healthz\": dial tcp 10.134.0.73:8643: connect: connection refused"
Sep 17 19:57:51.129117 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:51.129081    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 17 19:57:53.016810 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.016780    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:57:53.163395 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.163328    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c3d388-0def-4829-b231-2c8db4becaa2-kserve-provision-location\") pod \"86c3d388-0def-4829-b231-2c8db4becaa2\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") "
Sep 17 19:57:53.163548 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.163404    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-kube-rbac-proxy-sar-config\") pod \"86c3d388-0def-4829-b231-2c8db4becaa2\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") "
Sep 17 19:57:53.163548 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.163437    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-cabundle-cert\") pod \"86c3d388-0def-4829-b231-2c8db4becaa2\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") "
Sep 17 19:57:53.163548 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.163478    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c3d388-0def-4829-b231-2c8db4becaa2-proxy-tls\") pod \"86c3d388-0def-4829-b231-2c8db4becaa2\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") "
Sep 17 19:57:53.163548 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.163532    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fpmv\" (UniqueName: \"kubernetes.io/projected/86c3d388-0def-4829-b231-2c8db4becaa2-kube-api-access-4fpmv\") pod \"86c3d388-0def-4829-b231-2c8db4becaa2\" (UID: \"86c3d388-0def-4829-b231-2c8db4becaa2\") "
Sep 17 19:57:53.163816 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.163691    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c3d388-0def-4829-b231-2c8db4becaa2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86c3d388-0def-4829-b231-2c8db4becaa2" (UID: "86c3d388-0def-4829-b231-2c8db4becaa2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:57:53.163816 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.163777    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86c3d388-0def-4829-b231-2c8db4becaa2" (UID: "86c3d388-0def-4829-b231-2c8db4becaa2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:57:53.163905 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.163821    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86c3d388-0def-4829-b231-2c8db4becaa2-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:57:53.163905 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.163832    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86c3d388-0def-4829-b231-2c8db4becaa2" (UID: "86c3d388-0def-4829-b231-2c8db4becaa2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:57:53.165876 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.165856    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c3d388-0def-4829-b231-2c8db4becaa2-kube-api-access-4fpmv" (OuterVolumeSpecName: "kube-api-access-4fpmv") pod "86c3d388-0def-4829-b231-2c8db4becaa2" (UID: "86c3d388-0def-4829-b231-2c8db4becaa2"). InnerVolumeSpecName "kube-api-access-4fpmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:57:53.165936 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.165904    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c3d388-0def-4829-b231-2c8db4becaa2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86c3d388-0def-4829-b231-2c8db4becaa2" (UID: "86c3d388-0def-4829-b231-2c8db4becaa2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:57:53.264483 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.264447    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpmv\" (UniqueName: \"kubernetes.io/projected/86c3d388-0def-4829-b231-2c8db4becaa2-kube-api-access-4fpmv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:57:53.264483 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.264480    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:57:53.264483 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.264490    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86c3d388-0def-4829-b231-2c8db4becaa2-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:57:53.264694 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.264500    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86c3d388-0def-4829-b231-2c8db4becaa2-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:57:53.528448 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.528416    2583 generic.go:356] "Generic (PLEG): container finished" podID="86c3d388-0def-4829-b231-2c8db4becaa2" containerID="e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0" exitCode=0
Sep 17 19:57:53.528648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.528479    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"
Sep 17 19:57:53.528648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.528499    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" event={"ID":"86c3d388-0def-4829-b231-2c8db4becaa2","Type":"ContainerDied","Data":"e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0"}
Sep 17 19:57:53.528648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.528558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh" event={"ID":"86c3d388-0def-4829-b231-2c8db4becaa2","Type":"ContainerDied","Data":"d90d4827853cdd6246630e70b5e17f03e9fd392701cd366424699de247be8955"}
Sep 17 19:57:53.528648 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.528585    2583 scope.go:117] "RemoveContainer" containerID="221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f"
Sep 17 19:57:53.537916 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.537899    2583 scope.go:117] "RemoveContainer" containerID="e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0"
Sep 17 19:57:53.545789 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.545768    2583 scope.go:117] "RemoveContainer" containerID="bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4"
Sep 17 19:57:53.551255 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.551228    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"]
Sep 17 19:57:53.554364 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.554338    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-td4bh"]
Sep 17 19:57:53.555111 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.555094    2583 scope.go:117] "RemoveContainer" containerID="221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f"
Sep 17 19:57:53.555447 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:57:53.555426    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f\": container with ID starting with 221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f not found: ID does not exist" containerID="221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f"
Sep 17 19:57:53.555517 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.555460    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f"} err="failed to get container status \"221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f\": rpc error: code = NotFound desc = could not find container \"221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f\": container with ID starting with 221845c30440ed9044813de80a751bbd1f49fd639d09497897ce9c7a32aa817f not found: ID does not exist"
Sep 17 19:57:53.555517 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.555487    2583 scope.go:117] "RemoveContainer" containerID="e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0"
Sep 17 19:57:53.555817 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:57:53.555798    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0\": container with ID starting with e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0 not found: ID does not exist" containerID="e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0"
Sep 17 19:57:53.555862 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.555827    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0"} err="failed to get container status \"e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0\": rpc error: code = NotFound desc = could not find container \"e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0\": container with ID starting with e2e9da032913e262aeabced451cc1bc5b1cdadf54310d20908bdea742da9f3e0 not found: ID does not exist"
Sep 17 19:57:53.555862 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.555845    2583 scope.go:117] "RemoveContainer" containerID="bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4"
Sep 17 19:57:53.556083 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:57:53.556065    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4\": container with ID starting with bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4 not found: ID does not exist" containerID="bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4"
Sep 17 19:57:53.556128 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:53.556091    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4"} err="failed to get container status \"bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4\": rpc error: code = NotFound desc = could not find container \"bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4\": container with ID starting with bd8e489f395e0e0f75cf4bac20c1548fb8c3725abdb1248aeeaef80005e85de4 not found: ID does not exist"
Sep 17 19:57:54.200251 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.200217    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"]
Sep 17 19:57:54.200667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.200540    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="storage-initializer"
Sep 17 19:57:54.200667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.200552    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="storage-initializer"
Sep 17 19:57:54.200667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.200565    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kube-rbac-proxy"
Sep 17 19:57:54.200667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.200570    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kube-rbac-proxy"
Sep 17 19:57:54.200667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.200577    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container"
Sep 17 19:57:54.200667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.200582    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container"
Sep 17 19:57:54.200667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.200643    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kserve-container"
Sep 17 19:57:54.200667 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.200654    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" containerName="kube-rbac-proxy"
Sep 17 19:57:54.204975 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.204955    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.207364 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.207343    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-bc5294-predictor-serving-cert\""
Sep 17 19:57:54.207486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.207373    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:57:54.207486 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.207348    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:57:54.207693 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.207674    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-bc5294-kube-rbac-proxy-sar-config\""
Sep 17 19:57:54.207693 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.207688    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:57:54.207812 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.207734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:57:54.208277 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.208260    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:57:54.216255 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.216235    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"]
Sep 17 19:57:54.373895 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.373866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27a7159-41dc-43da-8745-999d03050e50-proxy-tls\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.373895 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.373903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-kube-rbac-proxy-sar-config\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.374125 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.373943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtk5\" (UniqueName: \"kubernetes.io/projected/e27a7159-41dc-43da-8745-999d03050e50-kube-api-access-mxtk5\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.374125 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.374045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27a7159-41dc-43da-8745-999d03050e50-kserve-provision-location\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.374125 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.374088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-cabundle-cert\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.474941 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.474818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27a7159-41dc-43da-8745-999d03050e50-proxy-tls\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.474941 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.474877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-kube-rbac-proxy-sar-config\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.474941 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.474901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtk5\" (UniqueName: \"kubernetes.io/projected/e27a7159-41dc-43da-8745-999d03050e50-kube-api-access-mxtk5\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.475224 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.474960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27a7159-41dc-43da-8745-999d03050e50-kserve-provision-location\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.475224 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:57:54.474990    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-bc5294-predictor-serving-cert: secret "isvc-primary-bc5294-predictor-serving-cert" not found
Sep 17 19:57:54.475224 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.474993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-cabundle-cert\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.475224 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:57:54.475062    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e27a7159-41dc-43da-8745-999d03050e50-proxy-tls podName:e27a7159-41dc-43da-8745-999d03050e50 nodeName:}" failed. No retries permitted until 2026-09-17 19:57:54.975041455 +0000 UTC m=+3094.102782852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e27a7159-41dc-43da-8745-999d03050e50-proxy-tls") pod "isvc-primary-bc5294-predictor-749d9d6745-2656p" (UID: "e27a7159-41dc-43da-8745-999d03050e50") : secret "isvc-primary-bc5294-predictor-serving-cert" not found
Sep 17 19:57:54.475366 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.475348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27a7159-41dc-43da-8745-999d03050e50-kserve-provision-location\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.475572 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.475554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-kube-rbac-proxy-sar-config\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.475649 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.475611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-cabundle-cert\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.483787 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.483768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtk5\" (UniqueName: \"kubernetes.io/projected/e27a7159-41dc-43da-8745-999d03050e50-kube-api-access-mxtk5\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.979331 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.979296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27a7159-41dc-43da-8745-999d03050e50-proxy-tls\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:54.982017 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:54.981989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27a7159-41dc-43da-8745-999d03050e50-proxy-tls\") pod \"isvc-primary-bc5294-predictor-749d9d6745-2656p\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") " pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:55.038288 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:55.038258    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c3d388-0def-4829-b231-2c8db4becaa2" path="/var/lib/kubelet/pods/86c3d388-0def-4829-b231-2c8db4becaa2/volumes"
Sep 17 19:57:55.115044 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:55.115017    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:57:55.388665 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:55.388481    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"]
Sep 17 19:57:55.390968 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:57:55.390938    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27a7159_41dc_43da_8745_999d03050e50.slice/crio-f4972864f0c25ac8004d24cf5a84f854378b6d01553bef728cc6320540d12569 WatchSource:0}: Error finding container f4972864f0c25ac8004d24cf5a84f854378b6d01553bef728cc6320540d12569: Status 404 returned error can't find the container with id f4972864f0c25ac8004d24cf5a84f854378b6d01553bef728cc6320540d12569
Sep 17 19:57:55.540563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:55.540527    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" event={"ID":"e27a7159-41dc-43da-8745-999d03050e50","Type":"ContainerStarted","Data":"933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7"}
Sep 17 19:57:55.540767 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:55.540586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" event={"ID":"e27a7159-41dc-43da-8745-999d03050e50","Type":"ContainerStarted","Data":"f4972864f0c25ac8004d24cf5a84f854378b6d01553bef728cc6320540d12569"}
Sep 17 19:57:59.561446 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:59.561411    2583 generic.go:356] "Generic (PLEG): container finished" podID="e27a7159-41dc-43da-8745-999d03050e50" containerID="933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7" exitCode=0
Sep 17 19:57:59.561937 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:57:59.561490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" event={"ID":"e27a7159-41dc-43da-8745-999d03050e50","Type":"ContainerDied","Data":"933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7"}
Sep 17 19:58:00.568767 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:00.568730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" event={"ID":"e27a7159-41dc-43da-8745-999d03050e50","Type":"ContainerStarted","Data":"64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a"}
Sep 17 19:58:00.569200 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:00.568791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" event={"ID":"e27a7159-41dc-43da-8745-999d03050e50","Type":"ContainerStarted","Data":"2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b"}
Sep 17 19:58:00.569200 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:00.568972    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:58:00.590419 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:00.590371    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podStartSLOduration=2.643288676 podStartE2EDuration="6.59035507s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.947066394s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:58:00.588915716 +0000 UTC m=+3099.716657118" watchObservedRunningTime="2026-09-17 19:58:00.59035507 +0000 UTC m=+3099.718096470"
Sep 17 19:58:01.579645 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:01.579595    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:58:01.581035 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:01.581003    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 19:58:02.584156 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:02.584120    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 19:58:07.589085 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:07.589057    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:58:07.589572 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:07.589539    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 19:58:17.589530 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:17.589488    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 19:58:27.590340 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:27.590304    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 19:58:37.590065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:37.589984    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 19:58:47.589683 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:47.589640    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 19:58:57.589834 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:58:57.589797    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 19:59:07.590065 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:07.590021    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 19:59:17.590695 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:17.590666    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 19:59:39.397881 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.397841    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"]
Sep 17 19:59:39.401406 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.401386    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.403563 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.403539    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-bc5294\""
Sep 17 19:59:39.403738 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.403691    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-bc5294-kube-rbac-proxy-sar-config\""
Sep 17 19:59:39.403738 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.403691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-bc5294-predictor-serving-cert\""
Sep 17 19:59:39.403908 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.403701    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-bc5294-dockercfg-x78v8\""
Sep 17 19:59:39.412566 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.412538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"]
Sep 17 19:59:39.530303 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.530261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kserve-provision-location\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.530471 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.530318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-proxy-tls\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.530471 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.530381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-cabundle-cert\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.530471 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.530397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdsdw\" (UniqueName: \"kubernetes.io/projected/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-api-access-gdsdw\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.530471 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.530439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.631437 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.631404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-proxy-tls\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.631607 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.631451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-cabundle-cert\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.631607 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.631471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdsdw\" (UniqueName: \"kubernetes.io/projected/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-api-access-gdsdw\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.631607 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.631505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.631607 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.631547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kserve-provision-location\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.631607 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:59:39.631584    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-serving-cert: secret "isvc-secondary-bc5294-predictor-serving-cert" not found
Sep 17 19:59:39.631904 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:59:39.631705    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-proxy-tls podName:5a933709-9ab4-4cc7-a7fc-a968ff0552d0 nodeName:}" failed. No retries permitted until 2026-09-17 19:59:40.131683481 +0000 UTC m=+3199.259424868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-proxy-tls") pod "isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf" (UID: "5a933709-9ab4-4cc7-a7fc-a968ff0552d0") : secret "isvc-secondary-bc5294-predictor-serving-cert" not found
Sep 17 19:59:39.632014 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.631992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kserve-provision-location\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.632191 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.632176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-cabundle-cert\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.632228 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.632202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:39.641840 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:39.641823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdsdw\" (UniqueName: \"kubernetes.io/projected/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-api-access-gdsdw\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:40.136497 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:40.136461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-proxy-tls\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:40.139049 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:40.139027    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-proxy-tls\") pod \"isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") " pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:40.311870 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:40.311840    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:40.586560 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:40.586536    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"]
Sep 17 19:59:40.588543 ip-10-0-140-110 kubenswrapper[2583]: W0917 19:59:40.588512    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a933709_9ab4_4cc7_a7fc_a968ff0552d0.slice/crio-e013a63e85b565e87cc92e8300866a5a97f06fea0f06b9394e6f74fdf6c91552 WatchSource:0}: Error finding container e013a63e85b565e87cc92e8300866a5a97f06fea0f06b9394e6f74fdf6c91552: Status 404 returned error can't find the container with id e013a63e85b565e87cc92e8300866a5a97f06fea0f06b9394e6f74fdf6c91552
Sep 17 19:59:40.997567 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:40.997533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf" event={"ID":"5a933709-9ab4-4cc7-a7fc-a968ff0552d0","Type":"ContainerStarted","Data":"9612ecedaaf88e285ecb2ab6588b8b87723de64799bd639a28f217ce76787f8f"}
Sep 17 19:59:40.997751 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:40.997603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf" event={"ID":"5a933709-9ab4-4cc7-a7fc-a968ff0552d0","Type":"ContainerStarted","Data":"e013a63e85b565e87cc92e8300866a5a97f06fea0f06b9394e6f74fdf6c91552"}
Sep 17 19:59:45.018156 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:45.018127    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_5a933709-9ab4-4cc7-a7fc-a968ff0552d0/storage-initializer/0.log"
Sep 17 19:59:45.018530 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:45.018166    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0" containerID="9612ecedaaf88e285ecb2ab6588b8b87723de64799bd639a28f217ce76787f8f" exitCode=1
Sep 17 19:59:45.018530 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:45.018218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf" event={"ID":"5a933709-9ab4-4cc7-a7fc-a968ff0552d0","Type":"ContainerDied","Data":"9612ecedaaf88e285ecb2ab6588b8b87723de64799bd639a28f217ce76787f8f"}
Sep 17 19:59:46.024309 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:46.024281    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_5a933709-9ab4-4cc7-a7fc-a968ff0552d0/storage-initializer/0.log"
Sep 17 19:59:46.024731 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:46.024371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf" event={"ID":"5a933709-9ab4-4cc7-a7fc-a968ff0552d0","Type":"ContainerStarted","Data":"ff15a57352d3e4febf86a2cbbd8d4feef1aebabb76b1b13de58bd6fbec4c6645"}
Sep 17 19:59:50.042600 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:50.042575    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_5a933709-9ab4-4cc7-a7fc-a968ff0552d0/storage-initializer/1.log"
Sep 17 19:59:50.043018 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:50.043001    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_5a933709-9ab4-4cc7-a7fc-a968ff0552d0/storage-initializer/0.log"
Sep 17 19:59:50.043069 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:50.043030    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0" containerID="ff15a57352d3e4febf86a2cbbd8d4feef1aebabb76b1b13de58bd6fbec4c6645" exitCode=1
Sep 17 19:59:50.043111 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:50.043093    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf" event={"ID":"5a933709-9ab4-4cc7-a7fc-a968ff0552d0","Type":"ContainerDied","Data":"ff15a57352d3e4febf86a2cbbd8d4feef1aebabb76b1b13de58bd6fbec4c6645"}
Sep 17 19:59:50.043156 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:50.043148    2583 scope.go:117] "RemoveContainer" containerID="9612ecedaaf88e285ecb2ab6588b8b87723de64799bd639a28f217ce76787f8f"
Sep 17 19:59:50.043499 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:50.043477    2583 scope.go:117] "RemoveContainer" containerID="9612ecedaaf88e285ecb2ab6588b8b87723de64799bd639a28f217ce76787f8f"
Sep 17 19:59:50.056252 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:59:50.056228    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_kserve-ci-e2e-test(5a933709-9ab4-4cc7-a7fc-a968ff0552d0)\"" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf" podUID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0"
Sep 17 19:59:50.056457 ip-10-0-140-110 kubenswrapper[2583]: E0917 19:59:50.056434    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_kserve-ci-e2e-test_5a933709-9ab4-4cc7-a7fc-a968ff0552d0_0 in pod sandbox e013a63e85b565e87cc92e8300866a5a97f06fea0f06b9394e6f74fdf6c91552 from index: no such id: '9612ecedaaf88e285ecb2ab6588b8b87723de64799bd639a28f217ce76787f8f'" containerID="9612ecedaaf88e285ecb2ab6588b8b87723de64799bd639a28f217ce76787f8f"
Sep 17 19:59:50.056527 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:50.056466    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9612ecedaaf88e285ecb2ab6588b8b87723de64799bd639a28f217ce76787f8f"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_kserve-ci-e2e-test_5a933709-9ab4-4cc7-a7fc-a968ff0552d0_0 in pod sandbox e013a63e85b565e87cc92e8300866a5a97f06fea0f06b9394e6f74fdf6c91552 from index: no such id: '9612ecedaaf88e285ecb2ab6588b8b87723de64799bd639a28f217ce76787f8f'"
Sep 17 19:59:51.048779 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:51.048751    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_5a933709-9ab4-4cc7-a7fc-a968ff0552d0/storage-initializer/1.log"
Sep 17 19:59:55.406468 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.406435    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"]
Sep 17 19:59:55.440599 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.440558    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"]
Sep 17 19:59:55.440887 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.440837    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" containerID="cri-o://2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b" gracePeriod=30
Sep 17 19:59:55.440990 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.440861    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kube-rbac-proxy" containerID="cri-o://64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a" gracePeriod=30
Sep 17 19:59:55.560370 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.560352    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_5a933709-9ab4-4cc7-a7fc-a968ff0552d0/storage-initializer/1.log"
Sep 17 19:59:55.562424 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.562408    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:55.659892 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.659816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kserve-provision-location\") pod \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") "
Sep 17 19:59:55.659892 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.659845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-proxy-tls\") pod \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") "
Sep 17 19:59:55.660099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.659900    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdsdw\" (UniqueName: \"kubernetes.io/projected/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-api-access-gdsdw\") pod \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") "
Sep 17 19:59:55.660099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.660015    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-cabundle-cert\") pod \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") "
Sep 17 19:59:55.660099 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.660062    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-rbac-proxy-sar-config\") pod \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\" (UID: \"5a933709-9ab4-4cc7-a7fc-a968ff0552d0\") "
Sep 17 19:59:55.660246 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.660133    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a933709-9ab4-4cc7-a7fc-a968ff0552d0" (UID: "5a933709-9ab4-4cc7-a7fc-a968ff0552d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:55.660392 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.660364    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a933709-9ab4-4cc7-a7fc-a968ff0552d0" (UID: "5a933709-9ab4-4cc7-a7fc-a968ff0552d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:59:55.660392 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.660391    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:59:55.660575 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.660396    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a933709-9ab4-4cc7-a7fc-a968ff0552d0" (UID: "5a933709-9ab4-4cc7-a7fc-a968ff0552d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:59:55.662145 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.662118    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a933709-9ab4-4cc7-a7fc-a968ff0552d0" (UID: "5a933709-9ab4-4cc7-a7fc-a968ff0552d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:59:55.662234 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.662189    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-api-access-gdsdw" (OuterVolumeSpecName: "kube-api-access-gdsdw") pod "5a933709-9ab4-4cc7-a7fc-a968ff0552d0" (UID: "5a933709-9ab4-4cc7-a7fc-a968ff0552d0"). InnerVolumeSpecName "kube-api-access-gdsdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:59:55.761324 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.761289    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:59:55.761324 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.761317    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdsdw\" (UniqueName: \"kubernetes.io/projected/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-api-access-gdsdw\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:59:55.761324 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.761326    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:59:55.761541 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:55.761335    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a933709-9ab4-4cc7-a7fc-a968ff0552d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 19:59:56.071056 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:56.071019    2583 generic.go:356] "Generic (PLEG): container finished" podID="e27a7159-41dc-43da-8745-999d03050e50" containerID="64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a" exitCode=2
Sep 17 19:59:56.071225 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:56.071086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" event={"ID":"e27a7159-41dc-43da-8745-999d03050e50","Type":"ContainerDied","Data":"64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a"}
Sep 17 19:59:56.072421 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:56.072395    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf_5a933709-9ab4-4cc7-a7fc-a968ff0552d0/storage-initializer/1.log"
Sep 17 19:59:56.072508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:56.072492    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"
Sep 17 19:59:56.072508 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:56.072501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf" event={"ID":"5a933709-9ab4-4cc7-a7fc-a968ff0552d0","Type":"ContainerDied","Data":"e013a63e85b565e87cc92e8300866a5a97f06fea0f06b9394e6f74fdf6c91552"}
Sep 17 19:59:56.072598 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:56.072546    2583 scope.go:117] "RemoveContainer" containerID="ff15a57352d3e4febf86a2cbbd8d4feef1aebabb76b1b13de58bd6fbec4c6645"
Sep 17 19:59:56.107045 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:56.107022    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"]
Sep 17 19:59:56.110963 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:56.110942    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc5294-predictor-79db8b4b7d-g8vjf"]
Sep 17 19:59:57.038043 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:57.038009    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0" path="/var/lib/kubelet/pods/5a933709-9ab4-4cc7-a7fc-a968ff0552d0/volumes"
Sep 17 19:59:57.584823 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:57.584787    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.74:8643/healthz\": dial tcp 10.134.0.74:8643: connect: connection refused"
Sep 17 19:59:57.589559 ip-10-0-140-110 kubenswrapper[2583]: I0917 19:59:57.589531    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 17 20:00:00.283462 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.283437    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 20:00:00.299508 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.299485    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27a7159-41dc-43da-8745-999d03050e50-proxy-tls\") pod \"e27a7159-41dc-43da-8745-999d03050e50\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") "
Sep 17 20:00:00.299624 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.299547    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27a7159-41dc-43da-8745-999d03050e50-kserve-provision-location\") pod \"e27a7159-41dc-43da-8745-999d03050e50\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") "
Sep 17 20:00:00.299624 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.299569    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-cabundle-cert\") pod \"e27a7159-41dc-43da-8745-999d03050e50\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") "
Sep 17 20:00:00.299624 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.299596    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxtk5\" (UniqueName: \"kubernetes.io/projected/e27a7159-41dc-43da-8745-999d03050e50-kube-api-access-mxtk5\") pod \"e27a7159-41dc-43da-8745-999d03050e50\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") "
Sep 17 20:00:00.299800 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.299683    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-kube-rbac-proxy-sar-config\") pod \"e27a7159-41dc-43da-8745-999d03050e50\" (UID: \"e27a7159-41dc-43da-8745-999d03050e50\") "
Sep 17 20:00:00.299985 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.299897    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27a7159-41dc-43da-8745-999d03050e50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e27a7159-41dc-43da-8745-999d03050e50" (UID: "e27a7159-41dc-43da-8745-999d03050e50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:00.300094 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.300000    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e27a7159-41dc-43da-8745-999d03050e50" (UID: "e27a7159-41dc-43da-8745-999d03050e50"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:00:00.300145 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.300082    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e27a7159-41dc-43da-8745-999d03050e50" (UID: "e27a7159-41dc-43da-8745-999d03050e50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:00:00.301590 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.301568    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27a7159-41dc-43da-8745-999d03050e50-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e27a7159-41dc-43da-8745-999d03050e50" (UID: "e27a7159-41dc-43da-8745-999d03050e50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:00:00.301832 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.301809    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27a7159-41dc-43da-8745-999d03050e50-kube-api-access-mxtk5" (OuterVolumeSpecName: "kube-api-access-mxtk5") pod "e27a7159-41dc-43da-8745-999d03050e50" (UID: "e27a7159-41dc-43da-8745-999d03050e50"). InnerVolumeSpecName "kube-api-access-mxtk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:00:00.400535 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.400467    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 20:00:00.400535 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.400493    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e27a7159-41dc-43da-8745-999d03050e50-proxy-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 20:00:00.400535 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.400503    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e27a7159-41dc-43da-8745-999d03050e50-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 20:00:00.400535 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.400512    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e27a7159-41dc-43da-8745-999d03050e50-cabundle-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 20:00:00.400535 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:00.400520    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtk5\" (UniqueName: \"kubernetes.io/projected/e27a7159-41dc-43da-8745-999d03050e50-kube-api-access-mxtk5\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 20:00:01.097134 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.097100    2583 generic.go:356] "Generic (PLEG): container finished" podID="e27a7159-41dc-43da-8745-999d03050e50" containerID="2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b" exitCode=0
Sep 17 20:00:01.097303 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.097167    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"
Sep 17 20:00:01.097303 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.097169    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" event={"ID":"e27a7159-41dc-43da-8745-999d03050e50","Type":"ContainerDied","Data":"2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b"}
Sep 17 20:00:01.097303 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.097296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p" event={"ID":"e27a7159-41dc-43da-8745-999d03050e50","Type":"ContainerDied","Data":"f4972864f0c25ac8004d24cf5a84f854378b6d01553bef728cc6320540d12569"}
Sep 17 20:00:01.097529 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.097330    2583 scope.go:117] "RemoveContainer" containerID="64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a"
Sep 17 20:00:01.105643 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.105597    2583 scope.go:117] "RemoveContainer" containerID="2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b"
Sep 17 20:00:01.112899 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.112881    2583 scope.go:117] "RemoveContainer" containerID="933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7"
Sep 17 20:00:01.115832 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.115809    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"]
Sep 17 20:00:01.119326 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.119304    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc5294-predictor-749d9d6745-2656p"]
Sep 17 20:00:01.121192 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.121176    2583 scope.go:117] "RemoveContainer" containerID="64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a"
Sep 17 20:00:01.121473 ip-10-0-140-110 kubenswrapper[2583]: E0917 20:00:01.121454    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a\": container with ID starting with 64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a not found: ID does not exist" containerID="64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a"
Sep 17 20:00:01.121513 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.121481    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a"} err="failed to get container status \"64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a\": rpc error: code = NotFound desc = could not find container \"64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a\": container with ID starting with 64924080f71a0003ac304e66826eeb720fa615e719ce6a2dbfa021725ffc621a not found: ID does not exist"
Sep 17 20:00:01.121513 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.121497    2583 scope.go:117] "RemoveContainer" containerID="2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b"
Sep 17 20:00:01.121758 ip-10-0-140-110 kubenswrapper[2583]: E0917 20:00:01.121741    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b\": container with ID starting with 2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b not found: ID does not exist" containerID="2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b"
Sep 17 20:00:01.121807 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.121764    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b"} err="failed to get container status \"2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b\": rpc error: code = NotFound desc = could not find container \"2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b\": container with ID starting with 2055554f0362c7662ffad0100490396a049163eac3925f2e9e2208bb2d00d80b not found: ID does not exist"
Sep 17 20:00:01.121807 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.121779    2583 scope.go:117] "RemoveContainer" containerID="933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7"
Sep 17 20:00:01.122054 ip-10-0-140-110 kubenswrapper[2583]: E0917 20:00:01.122035    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7\": container with ID starting with 933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7 not found: ID does not exist" containerID="933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7"
Sep 17 20:00:01.122095 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.122059    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7"} err="failed to get container status \"933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7\": rpc error: code = NotFound desc = could not find container \"933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7\": container with ID starting with 933d2855d78d54d5238c051110630fb07c1338c649fb7e6761b2effd4f22a0f7 not found: ID does not exist"
Sep 17 20:00:01.946254 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946222    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wxgh2/must-gather-9kh72"]
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946529    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="storage-initializer"
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946540    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="storage-initializer"
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946548    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0" containerName="storage-initializer"
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946554    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0" containerName="storage-initializer"
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946566    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container"
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946573    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container"
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946582    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kube-rbac-proxy"
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946587    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kube-rbac-proxy"
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946593    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0" containerName="storage-initializer"
Sep 17 20:00:01.946633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946598    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0" containerName="storage-initializer"
Sep 17 20:00:01.946942 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946670    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0" containerName="storage-initializer"
Sep 17 20:00:01.946942 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946677    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kserve-container"
Sep 17 20:00:01.946942 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946685    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27a7159-41dc-43da-8745-999d03050e50" containerName="kube-rbac-proxy"
Sep 17 20:00:01.946942 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.946771    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a933709-9ab4-4cc7-a7fc-a968ff0552d0" containerName="storage-initializer"
Sep 17 20:00:01.951026 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.951006    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:01.956173 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.956136    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wxgh2\"/\"kube-root-ca.crt\""
Sep 17 20:00:01.956286 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.956217    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wxgh2\"/\"openshift-service-ca.crt\""
Sep 17 20:00:01.964454 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:01.964431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wxgh2/must-gather-9kh72"]
Sep 17 20:00:02.009725 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:02.009692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkldt\" (UniqueName: \"kubernetes.io/projected/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-kube-api-access-rkldt\") pod \"must-gather-9kh72\" (UID: \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\") " pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:02.009888 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:02.009790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-must-gather-output\") pod \"must-gather-9kh72\" (UID: \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\") " pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:02.110412 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:02.110380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkldt\" (UniqueName: \"kubernetes.io/projected/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-kube-api-access-rkldt\") pod \"must-gather-9kh72\" (UID: \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\") " pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:02.110717 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:02.110474    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-must-gather-output\") pod \"must-gather-9kh72\" (UID: \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\") " pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:02.110854 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:02.110833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-must-gather-output\") pod \"must-gather-9kh72\" (UID: \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\") " pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:02.119609 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:02.119585    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkldt\" (UniqueName: \"kubernetes.io/projected/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-kube-api-access-rkldt\") pod \"must-gather-9kh72\" (UID: \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\") " pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:02.278569 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:02.278434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:02.552118 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:02.552092    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wxgh2/must-gather-9kh72"]
Sep 17 20:00:02.554792 ip-10-0-140-110 kubenswrapper[2583]: W0917 20:00:02.554766    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d3161a9_fc8d_4c63_a6d1_7b63eabdae12.slice/crio-91a8b582133888befbde955debc987fa9928f1514f6248bdec0d7189888331a5 WatchSource:0}: Error finding container 91a8b582133888befbde955debc987fa9928f1514f6248bdec0d7189888331a5: Status 404 returned error can't find the container with id 91a8b582133888befbde955debc987fa9928f1514f6248bdec0d7189888331a5
Sep 17 20:00:02.556725 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:02.556698    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:00:03.038081 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:03.038044    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e27a7159-41dc-43da-8745-999d03050e50" path="/var/lib/kubelet/pods/e27a7159-41dc-43da-8745-999d03050e50/volumes"
Sep 17 20:00:03.106941 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:03.106911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxgh2/must-gather-9kh72" event={"ID":"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12","Type":"ContainerStarted","Data":"91a8b582133888befbde955debc987fa9928f1514f6248bdec0d7189888331a5"}
Sep 17 20:00:10.147628 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:10.147523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxgh2/must-gather-9kh72" event={"ID":"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12","Type":"ContainerStarted","Data":"4af87f825d96380ec03fcebbd9d5c52bf1eefb90a6b2d45ec2d407689d04c71d"}
Sep 17 20:00:10.147628 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:10.147567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxgh2/must-gather-9kh72" event={"ID":"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12","Type":"ContainerStarted","Data":"59dd3fbcc652dbac2063f19ef24fdc0a2cab1d76fcd9c9da8576c8d1f84114fd"}
Sep 17 20:00:10.164309 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:10.163941    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wxgh2/must-gather-9kh72" podStartSLOduration=1.916684197 podStartE2EDuration="9.16391846s" totalImagesPullingTime="7.247234263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:00:10.162234603 +0000 UTC m=+3229.289976015" watchObservedRunningTime="2026-09-17 20:00:10.16391846 +0000 UTC m=+3229.291659863"
Sep 17 20:00:38.273306 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:38.273270    2583 generic.go:356] "Generic (PLEG): container finished" podID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" containerID="59dd3fbcc652dbac2063f19ef24fdc0a2cab1d76fcd9c9da8576c8d1f84114fd" exitCode=0
Sep 17 20:00:38.273859 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:38.273356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxgh2/must-gather-9kh72" event={"ID":"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12","Type":"ContainerDied","Data":"59dd3fbcc652dbac2063f19ef24fdc0a2cab1d76fcd9c9da8576c8d1f84114fd"}
Sep 17 20:00:38.273859 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:38.273814    2583 scope.go:117] "RemoveContainer" containerID="59dd3fbcc652dbac2063f19ef24fdc0a2cab1d76fcd9c9da8576c8d1f84114fd"
Sep 17 20:00:38.597647 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:38.597543    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wxgh2_must-gather-9kh72_5d3161a9-fc8d-4c63-a6d1-7b63eabdae12/gather/0.log"
Sep 17 20:00:39.310084 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.310053    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rdtgx/must-gather-c4pfv"]
Sep 17 20:00:39.312487 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.312463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdtgx/must-gather-c4pfv"
Sep 17 20:00:39.314995 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.314968    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rdtgx\"/\"default-dockercfg-bqm8w\""
Sep 17 20:00:39.315123 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.315015    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rdtgx\"/\"kube-root-ca.crt\""
Sep 17 20:00:39.315818 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.315801    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rdtgx\"/\"openshift-service-ca.crt\""
Sep 17 20:00:39.321365 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.321342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdtgx/must-gather-c4pfv"]
Sep 17 20:00:39.454191 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.454144    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b102efe8-a2ed-4787-a663-ac7e9d22c439-must-gather-output\") pod \"must-gather-c4pfv\" (UID: \"b102efe8-a2ed-4787-a663-ac7e9d22c439\") " pod="openshift-must-gather-rdtgx/must-gather-c4pfv"
Sep 17 20:00:39.454380 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.454203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rkxp\" (UniqueName: \"kubernetes.io/projected/b102efe8-a2ed-4787-a663-ac7e9d22c439-kube-api-access-9rkxp\") pod \"must-gather-c4pfv\" (UID: \"b102efe8-a2ed-4787-a663-ac7e9d22c439\") " pod="openshift-must-gather-rdtgx/must-gather-c4pfv"
Sep 17 20:00:39.555164 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.555129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b102efe8-a2ed-4787-a663-ac7e9d22c439-must-gather-output\") pod \"must-gather-c4pfv\" (UID: \"b102efe8-a2ed-4787-a663-ac7e9d22c439\") " pod="openshift-must-gather-rdtgx/must-gather-c4pfv"
Sep 17 20:00:39.555296 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.555189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rkxp\" (UniqueName: \"kubernetes.io/projected/b102efe8-a2ed-4787-a663-ac7e9d22c439-kube-api-access-9rkxp\") pod \"must-gather-c4pfv\" (UID: \"b102efe8-a2ed-4787-a663-ac7e9d22c439\") " pod="openshift-must-gather-rdtgx/must-gather-c4pfv"
Sep 17 20:00:39.555472 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.555457    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b102efe8-a2ed-4787-a663-ac7e9d22c439-must-gather-output\") pod \"must-gather-c4pfv\" (UID: \"b102efe8-a2ed-4787-a663-ac7e9d22c439\") " pod="openshift-must-gather-rdtgx/must-gather-c4pfv"
Sep 17 20:00:39.563942 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.563882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rkxp\" (UniqueName: \"kubernetes.io/projected/b102efe8-a2ed-4787-a663-ac7e9d22c439-kube-api-access-9rkxp\") pod \"must-gather-c4pfv\" (UID: \"b102efe8-a2ed-4787-a663-ac7e9d22c439\") " pod="openshift-must-gather-rdtgx/must-gather-c4pfv"
Sep 17 20:00:39.622849 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.622816    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdtgx/must-gather-c4pfv"
Sep 17 20:00:39.893094 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:39.893015    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdtgx/must-gather-c4pfv"]
Sep 17 20:00:39.895393 ip-10-0-140-110 kubenswrapper[2583]: W0917 20:00:39.895363    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb102efe8_a2ed_4787_a663_ac7e9d22c439.slice/crio-0c1d329c8347f32b4e0a6ca38810567935bfeeda1b51d8ef62726789f4899e24 WatchSource:0}: Error finding container 0c1d329c8347f32b4e0a6ca38810567935bfeeda1b51d8ef62726789f4899e24: Status 404 returned error can't find the container with id 0c1d329c8347f32b4e0a6ca38810567935bfeeda1b51d8ef62726789f4899e24
Sep 17 20:00:40.282502 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:40.282469    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdtgx/must-gather-c4pfv" event={"ID":"b102efe8-a2ed-4787-a663-ac7e9d22c439","Type":"ContainerStarted","Data":"0c1d329c8347f32b4e0a6ca38810567935bfeeda1b51d8ef62726789f4899e24"}
Sep 17 20:00:41.291176 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:41.291128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdtgx/must-gather-c4pfv" event={"ID":"b102efe8-a2ed-4787-a663-ac7e9d22c439","Type":"ContainerStarted","Data":"3ae43d0f1347123abbb8e7c89881b110fd4fe5f71c192b120baa28bad77973cd"}
Sep 17 20:00:41.291683 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:41.291193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdtgx/must-gather-c4pfv" event={"ID":"b102efe8-a2ed-4787-a663-ac7e9d22c439","Type":"ContainerStarted","Data":"a1f1e0ca41aa54301d33f73b2c55a924767a83f03be1bbc79e16f50cdda02da8"}
Sep 17 20:00:41.310058 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:41.309999    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rdtgx/must-gather-c4pfv" podStartSLOduration=1.614987711 podStartE2EDuration="2.309979703s" totalImagesPullingTime="694.991992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:00:41.30866417 +0000 UTC m=+3260.436405573" watchObservedRunningTime="2026-09-17 20:00:41.309979703 +0000 UTC m=+3260.437721107"
Sep 17 20:00:42.355492 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:42.355447    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xhl95_b5e92de3-419a-4bff-858a-d15f9509c77a/global-pull-secret-syncer/0.log"
Sep 17 20:00:42.450379 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:42.450343    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s7wdr_4afed170-1f22-445b-b281-d4534939b551/konnectivity-agent/0.log"
Sep 17 20:00:42.570528 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:42.570498    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-110.ec2.internal_4eb197abc8b394d582a58b08fa210ad1/haproxy/0.log"
Sep 17 20:00:44.165820 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.163048    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-wxgh2/must-gather-9kh72"]
Sep 17 20:00:44.166325 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.165864    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-wxgh2/must-gather-9kh72"]
Sep 17 20:00:44.166325 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.166043    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-wxgh2/must-gather-9kh72" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" containerName="copy" containerID="cri-o://4af87f825d96380ec03fcebbd9d5c52bf1eefb90a6b2d45ec2d407689d04c71d" gracePeriod=2
Sep 17 20:00:44.168950 ip-10-0-140-110 kubenswrapper[2583]: E0917 20:00:44.168900    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9kh72\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wxgh2\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:44.324291 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.312605    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wxgh2_must-gather-9kh72_5d3161a9-fc8d-4c63-a6d1-7b63eabdae12/copy/0.log"
Sep 17 20:00:44.324291 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.314772    2583 generic.go:356] "Generic (PLEG): container finished" podID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" containerID="4af87f825d96380ec03fcebbd9d5c52bf1eefb90a6b2d45ec2d407689d04c71d" exitCode=143
Sep 17 20:00:44.549660 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.549191    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wxgh2_must-gather-9kh72_5d3161a9-fc8d-4c63-a6d1-7b63eabdae12/copy/0.log"
Sep 17 20:00:44.553019 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.552768    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:44.558307 ip-10-0-140-110 kubenswrapper[2583]: E0917 20:00:44.558258    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9kh72\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wxgh2\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:44.604163 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.604105    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-must-gather-output\") pod \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\" (UID: \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\") "
Sep 17 20:00:44.605954 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.605917    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" (UID: "5d3161a9-fc8d-4c63-a6d1-7b63eabdae12"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:44.705940 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.705022    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkldt\" (UniqueName: \"kubernetes.io/projected/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-kube-api-access-rkldt\") pod \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\" (UID: \"5d3161a9-fc8d-4c63-a6d1-7b63eabdae12\") "
Sep 17 20:00:44.705940 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.705321    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-must-gather-output\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 20:00:44.708257 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.708211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-kube-api-access-rkldt" (OuterVolumeSpecName: "kube-api-access-rkldt") pod "5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" (UID: "5d3161a9-fc8d-4c63-a6d1-7b63eabdae12"). InnerVolumeSpecName "kube-api-access-rkldt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:00:44.806556 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:44.806464    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkldt\" (UniqueName: \"kubernetes.io/projected/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12-kube-api-access-rkldt\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Sep 17 20:00:45.043788 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:45.042381    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" path="/var/lib/kubelet/pods/5d3161a9-fc8d-4c63-a6d1-7b63eabdae12/volumes"
Sep 17 20:00:45.323589 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:45.323170    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wxgh2_must-gather-9kh72_5d3161a9-fc8d-4c63-a6d1-7b63eabdae12/copy/0.log"
Sep 17 20:00:45.325546 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:45.324828    2583 scope.go:117] "RemoveContainer" containerID="4af87f825d96380ec03fcebbd9d5c52bf1eefb90a6b2d45ec2d407689d04c71d"
Sep 17 20:00:45.325546 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:45.325003    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxgh2/must-gather-9kh72"
Sep 17 20:00:45.353930 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:45.353895    2583 scope.go:117] "RemoveContainer" containerID="59dd3fbcc652dbac2063f19ef24fdc0a2cab1d76fcd9c9da8576c8d1f84114fd"
Sep 17 20:00:46.935790 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:46.935749    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_33731d5a-c3dc-4464-ba64-686c33118bd9/alertmanager/0.log"
Sep 17 20:00:46.963338 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:46.963302    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_33731d5a-c3dc-4464-ba64-686c33118bd9/config-reloader/0.log"
Sep 17 20:00:46.988590 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:46.988564    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_33731d5a-c3dc-4464-ba64-686c33118bd9/kube-rbac-proxy-web/0.log"
Sep 17 20:00:47.016701 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.016676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_33731d5a-c3dc-4464-ba64-686c33118bd9/kube-rbac-proxy/0.log"
Sep 17 20:00:47.058442 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.058406    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_33731d5a-c3dc-4464-ba64-686c33118bd9/kube-rbac-proxy-metric/0.log"
Sep 17 20:00:47.088276 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.088242    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_33731d5a-c3dc-4464-ba64-686c33118bd9/prom-label-proxy/0.log"
Sep 17 20:00:47.143463 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.143416    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_33731d5a-c3dc-4464-ba64-686c33118bd9/init-config-reloader/0.log"
Sep 17 20:00:47.220489 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.220367    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-zqfxg_6541cbbf-c6d5-49b1-9637-dfa61a479d57/kube-state-metrics/0.log"
Sep 17 20:00:47.239172 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.239139    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-zqfxg_6541cbbf-c6d5-49b1-9637-dfa61a479d57/kube-rbac-proxy-main/0.log"
Sep 17 20:00:47.263186 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.263157    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-zqfxg_6541cbbf-c6d5-49b1-9637-dfa61a479d57/kube-rbac-proxy-self/0.log"
Sep 17 20:00:47.324925 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.324886    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-nwjbm_1f9df7ef-ed1d-48a9-a8e7-cdcd44ac6063/monitoring-plugin/0.log"
Sep 17 20:00:47.518636 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.518529    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qhp6v_264b1a88-c09c-466e-b0c1-b5878a8c0184/node-exporter/0.log"
Sep 17 20:00:47.543238 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.543199    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qhp6v_264b1a88-c09c-466e-b0c1-b5878a8c0184/kube-rbac-proxy/0.log"
Sep 17 20:00:47.568213 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.567262    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qhp6v_264b1a88-c09c-466e-b0c1-b5878a8c0184/init-textfile/0.log"
Sep 17 20:00:47.600887 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.600842    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-n9q67_69e66a75-f4d6-4f86-ad91-31f4aebd0309/kube-rbac-proxy-main/0.log"
Sep 17 20:00:47.627227 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.627194    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-n9q67_69e66a75-f4d6-4f86-ad91-31f4aebd0309/kube-rbac-proxy-self/0.log"
Sep 17 20:00:47.651295 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.651265    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-n9q67_69e66a75-f4d6-4f86-ad91-31f4aebd0309/openshift-state-metrics/0.log"
Sep 17 20:00:47.699929 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.699893    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9ca9fe-6839-4aaf-808e-4ba059036268/prometheus/0.log"
Sep 17 20:00:47.719432 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.719399    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9ca9fe-6839-4aaf-808e-4ba059036268/config-reloader/0.log"
Sep 17 20:00:47.744818 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.744777    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9ca9fe-6839-4aaf-808e-4ba059036268/thanos-sidecar/0.log"
Sep 17 20:00:47.766849 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.766822    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9ca9fe-6839-4aaf-808e-4ba059036268/kube-rbac-proxy-web/0.log"
Sep 17 20:00:47.797497 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.797423    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9ca9fe-6839-4aaf-808e-4ba059036268/kube-rbac-proxy/0.log"
Sep 17 20:00:47.822982 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.822952    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9ca9fe-6839-4aaf-808e-4ba059036268/kube-rbac-proxy-thanos/0.log"
Sep 17 20:00:47.846924 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.846890    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9e9ca9fe-6839-4aaf-808e-4ba059036268/init-config-reloader/0.log"
Sep 17 20:00:47.879964 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.879932    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-4mf42_b5c9c5f4-869b-4b14-8f1c-1ab03e01b013/prometheus-operator/0.log"
Sep 17 20:00:47.899848 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:47.899817    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-4mf42_b5c9c5f4-869b-4b14-8f1c-1ab03e01b013/kube-rbac-proxy/0.log"
Sep 17 20:00:48.070320 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:48.070228    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5677f75b99-pz5zs_9c6ceaf7-95b8-41f7-812e-8f93c1636316/thanos-query/0.log"
Sep 17 20:00:48.108331 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:48.108291    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5677f75b99-pz5zs_9c6ceaf7-95b8-41f7-812e-8f93c1636316/kube-rbac-proxy-web/0.log"
Sep 17 20:00:48.139341 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:48.139310    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5677f75b99-pz5zs_9c6ceaf7-95b8-41f7-812e-8f93c1636316/kube-rbac-proxy/0.log"
Sep 17 20:00:48.172775 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:48.172746    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5677f75b99-pz5zs_9c6ceaf7-95b8-41f7-812e-8f93c1636316/prom-label-proxy/0.log"
Sep 17 20:00:48.205061 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:48.205028    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5677f75b99-pz5zs_9c6ceaf7-95b8-41f7-812e-8f93c1636316/kube-rbac-proxy-rules/0.log"
Sep 17 20:00:48.242019 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:48.241888    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5677f75b99-pz5zs_9c6ceaf7-95b8-41f7-812e-8f93c1636316/kube-rbac-proxy-metrics/0.log"
Sep 17 20:00:50.701838 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.701797    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"]
Sep 17 20:00:50.702429 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.702355    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" containerName="copy"
Sep 17 20:00:50.702429 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.702376    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" containerName="copy"
Sep 17 20:00:50.702429 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.702399    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" containerName="gather"
Sep 17 20:00:50.702429 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.702409    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" containerName="gather"
Sep 17 20:00:50.702711 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.702480    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" containerName="copy"
Sep 17 20:00:50.702711 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.702496    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3161a9-fc8d-4c63-a6d1-7b63eabdae12" containerName="gather"
Sep 17 20:00:50.706444 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.706417    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.720468 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.720416    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"]
Sep 17 20:00:50.756642 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.755467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-lib-modules\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.756642 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.755515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-podres\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.756642 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.755554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4p62\" (UniqueName: \"kubernetes.io/projected/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-kube-api-access-z4p62\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.756642 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.755630    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-proc\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.756642 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.755661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-sys\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.857186 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.857111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-lib-modules\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.857464 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.857442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-lib-modules\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.857670 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.857648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-podres\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.857783 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.857688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4p62\" (UniqueName: \"kubernetes.io/projected/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-kube-api-access-z4p62\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.857783 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.857727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-proc\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.857783 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.857754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-sys\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.857922 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.857823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-podres\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.857922 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.857852    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-proc\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.857922 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.857865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-sys\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:50.866822 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:50.866793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4p62\" (UniqueName: \"kubernetes.io/projected/b7ecfabe-1dd2-4eff-a394-0e54c8986b54-kube-api-access-z4p62\") pod \"perf-node-gather-daemonset-pnqlg\" (UID: \"b7ecfabe-1dd2-4eff-a394-0e54c8986b54\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:51.019383 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:51.019296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:51.312669 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:51.312567    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"]
Sep 17 20:00:51.363115 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:51.363085    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg" event={"ID":"b7ecfabe-1dd2-4eff-a394-0e54c8986b54","Type":"ContainerStarted","Data":"d71323b43e7cedba98fd75fb56cfa1c4b04d42d84ee3759fc0ca802b51213ac0"}
Sep 17 20:00:51.916958 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:51.916917    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jvm6_63c7b622-334d-451f-9a47-f3a10c8d7f9b/dns/0.log"
Sep 17 20:00:51.939877 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:51.939844    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jvm6_63c7b622-334d-451f-9a47-f3a10c8d7f9b/kube-rbac-proxy/0.log"
Sep 17 20:00:52.089514 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:52.089485    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jqm7n_7db784ac-d4f5-4f32-ab91-73e83ea92246/dns-node-resolver/0.log"
Sep 17 20:00:52.370342 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:52.370257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg" event={"ID":"b7ecfabe-1dd2-4eff-a394-0e54c8986b54","Type":"ContainerStarted","Data":"d74045c4830ef7bf54f1cd7c3bb06f29a1f5112350ee830d62f1bf50e0611784"}
Sep 17 20:00:52.370491 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:52.370359    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:00:52.388666 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:52.388592    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg" podStartSLOduration=2.388560053 podStartE2EDuration="2.388560053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:00:52.387213663 +0000 UTC m=+3271.514955061" watchObservedRunningTime="2026-09-17 20:00:52.388560053 +0000 UTC m=+3271.516301473"
Sep 17 20:00:52.635120 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:52.635034    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8fmwv_14e2fdfb-a0fe-4077-a5fa-3cbf0cd01151/node-ca/0.log"
Sep 17 20:00:53.784602 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:53.784570    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sn6l5_5cc5e562-8481-4944-a52b-e0e7f6e5b991/serve-healthcheck-canary/0.log"
Sep 17 20:00:54.303713 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:54.303677    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xf97s_5525c384-23ac-4ae2-8b01-35a234b1dee8/kube-rbac-proxy/0.log"
Sep 17 20:00:54.325163 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:54.325129    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xf97s_5525c384-23ac-4ae2-8b01-35a234b1dee8/exporter/0.log"
Sep 17 20:00:54.347576 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:54.347544    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xf97s_5525c384-23ac-4ae2-8b01-35a234b1dee8/extractor/0.log"
Sep 17 20:00:57.051817 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:57.051786    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kgjbc_674ff399-31c0-4719-a368-341bb74b9153/s3-init/0.log"
Sep 17 20:00:57.069122 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:57.069087    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kgjbc_674ff399-31c0-4719-a368-341bb74b9153/download-hf-model/0.log"
Sep 17 20:00:57.105125 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:57.105088    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-2lp4j_fd741661-cca6-484b-8133-10cebfd86a19/s3-tls-init-custom/0.log"
Sep 17 20:00:57.123168 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:57.123140    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-2lp4j_fd741661-cca6-484b-8133-10cebfd86a19/download-hf-model/0.log"
Sep 17 20:00:57.155587 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:57.155557    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-df4l7_31b8b2c8-3011-43e2-8b30-033db4c476a4/s3-tls-init-serving/0.log"
Sep 17 20:00:57.170945 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:57.170922    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-df4l7_31b8b2c8-3011-43e2-8b30-033db4c476a4/download-hf-model/0.log"
Sep 17 20:00:57.200418 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:57.200391    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2fmrq_5c4882cf-2553-458d-94df-0ffdce8f0427/seaweedfs/0.log"
Sep 17 20:00:58.384725 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:00:58.384695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-pnqlg"
Sep 17 20:01:02.903076 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:02.902982    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7mb94_9147622f-fd81-4ea0-a693-7e4d0dab9d0b/kube-multus/0.log"
Sep 17 20:01:03.248899 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:03.248867    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vp9lm_931f933b-ce87-44d1-94ac-8dce917c3546/kube-multus-additional-cni-plugins/0.log"
Sep 17 20:01:03.270147 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:03.270116    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vp9lm_931f933b-ce87-44d1-94ac-8dce917c3546/egress-router-binary-copy/0.log"
Sep 17 20:01:03.291452 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:03.291422    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vp9lm_931f933b-ce87-44d1-94ac-8dce917c3546/cni-plugins/0.log"
Sep 17 20:01:03.314942 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:03.314918    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vp9lm_931f933b-ce87-44d1-94ac-8dce917c3546/bond-cni-plugin/0.log"
Sep 17 20:01:03.342917 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:03.342888    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vp9lm_931f933b-ce87-44d1-94ac-8dce917c3546/routeoverride-cni/0.log"
Sep 17 20:01:03.365827 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:03.365800    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vp9lm_931f933b-ce87-44d1-94ac-8dce917c3546/whereabouts-cni-bincopy/0.log"
Sep 17 20:01:03.390438 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:03.390418    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vp9lm_931f933b-ce87-44d1-94ac-8dce917c3546/whereabouts-cni/0.log"
Sep 17 20:01:03.523987 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:03.523909    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g886c_5da89ee3-23c3-4a18-a046-0dde553add07/network-metrics-daemon/0.log"
Sep 17 20:01:03.539728 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:03.539701    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g886c_5da89ee3-23c3-4a18-a046-0dde553add07/kube-rbac-proxy/0.log"
Sep 17 20:01:04.824784 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:04.824753    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-controller/0.log"
Sep 17 20:01:04.843995 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:04.843965    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/0.log"
Sep 17 20:01:04.844165 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:04.844100    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovn-acl-logging/1.log"
Sep 17 20:01:04.863856 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:04.863834    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/kube-rbac-proxy-node/0.log"
Sep 17 20:01:04.885881 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:04.885851    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 20:01:04.907633 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:04.907595    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/northd/0.log"
Sep 17 20:01:04.929464 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:04.929436    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/nbdb/0.log"
Sep 17 20:01:04.967759 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:04.967738    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/sbdb/0.log"
Sep 17 20:01:05.037058 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:05.037027    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtf4n_9c0c7c61-2325-402e-a299-0b24efefe9fd/ovnkube-controller/0.log"
Sep 17 20:01:06.148942 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:06.148914    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vdcz8_abc81dc4-5b90-4a46-ad49-35d0fb866226/network-check-target-container/0.log"
Sep 17 20:01:07.115990 ip-10-0-140-110 kubenswrapper[2583]: I0917 20:01:07.115951    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g2dhw_156da910-4b26-4c06-83f9-aaea524edfbb/iptables-alerter/0.log"