Sep 18 20:18:27.147240 ip-10-0-143-76 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 20:18:27.147250 ip-10-0-143-76 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 20:18:27.147260 ip-10-0-143-76 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 20:18:27.147530 ip-10-0-143-76 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 20:18:37.232599 ip-10-0-143-76 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 20:18:37.232614 ip-10-0-143-76 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7121a7cd47fd4a828105f5ae0a6484a2 --
Sep 18 20:20:30.859298 ip-10-0-143-76 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 20:20:31.236124 ip-10-0-143-76 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:31.236124 ip-10-0-143-76 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 20:20:31.236124 ip-10-0-143-76 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:31.236124 ip-10-0-143-76 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 20:20:31.236124 ip-10-0-143-76 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 20:20:31.237848 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.237771    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 20:20:31.243769 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243755    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:31.243769 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243770    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243774    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243777    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243781    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243784    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243787    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243789    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243792    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243796    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243798    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243801    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243803    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243806    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243808    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243813    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243817    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243820    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243823    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243825    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243828    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:31.243833 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243830    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243833    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243836    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243839    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243842    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243845    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243847    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243850    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243852    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243855    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243857    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243860    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243862    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243865    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243867    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243870    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243872    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243875    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243878    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:31.244324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243880    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243883    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243886    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243888    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243890    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243893    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243895    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243898    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243900    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243903    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243905    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243907    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243910    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243912    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243914    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243918    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243921    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243924    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243927    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243929    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:31.244785 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243931    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243934    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243937    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243940    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243942    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243945    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243947    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243950    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243952    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243955    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243957    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243960    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243963    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243965    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243968    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243970    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243972    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243975    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243977    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243980    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:31.245290 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243982    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243984    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243989    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243993    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243996    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.243999    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244002    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244006    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244019    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244022    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244025    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244030    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244033    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244037    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244041    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244044    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244047    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244050    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244053    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:31.245773 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244746    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244752    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244755    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244758    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244761    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244766    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244770    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244773    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244775    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244778    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244781    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244783    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244786    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244789    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244792    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244795    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244798    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244800    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244803    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244805    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:31.246273 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244809    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244811    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244814    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244817    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244819    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244822    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244825    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244827    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244830    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244832    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244835    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244839    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244842    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244845    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244848    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244851    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244855    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244857    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244861    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244863    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:31.246768 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244866    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244868    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244871    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244874    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244876    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244879    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244882    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244885    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244887    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244890    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244892    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244895    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244898    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244901    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244905    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244908    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244911    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244914    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244916    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:31.247363 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244919    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244921    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244924    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244927    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244929    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244932    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244934    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244936    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244939    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244942    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244944    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244946    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244949    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244953    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244956    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244959    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244961    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244964    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244966    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244969    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:31.247832 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244971    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244974    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244978    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244981    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244983    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244986    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244988    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244991    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244994    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244996    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.244999    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245001    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245004    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245006    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245009    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245025    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245028    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245031    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245033    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:31.248338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245036    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245418    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245423    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245426    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245428    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245432    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245435    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245438    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245440    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245443    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245448    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245452    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245455    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245458    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245461    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245467    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245470    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245473    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245476    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:31.248803 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245479    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245481    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245484    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245487    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245490    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245493    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245496    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245498    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245501    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245504    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245507    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245510    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245513    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245515    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245518    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245520    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245523    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245525    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245528    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245531    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:31.249289 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245534    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245537    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245539    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245542    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245545    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245547    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245550    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245552    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245557    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245561    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245564    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245567    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245569    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245572    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245574    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245577    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245580    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245582    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245585    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:31.249770 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245587    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245590    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245593    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245595    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245598    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245600    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245603    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245605    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245608    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245610    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245613    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245616    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245618    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245624    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245627    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245630    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245632    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245635    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245637    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245640    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:31.250244 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245642    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245646    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245649    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245652    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245655    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245658    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245660    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245663    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245665    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245668    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245670    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245673    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245676    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245678    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245681    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245683    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245686    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245688    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245691    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245693    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:31.250726 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245696    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.245699    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245766    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245798    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245805    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245810    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245820    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245823    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245827    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245832    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245835    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245838    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245841    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245846    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245849    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245852    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245855    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245858    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245861    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245864    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245868    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245871    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245874    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245877    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 20:20:31.251248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245881    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245885    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245888    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245907    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245911    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245914    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245918    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245921    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245925    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245928    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245932    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245935    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245938    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245942    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245945    2566 flags.go:64] FLAG: --enable-server="true"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245957    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245962    2566 flags.go:64] FLAG: --event-burst="100"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245966    2566 flags.go:64] FLAG: --event-qps="50"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245969    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245972    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245975    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245984    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245987    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245990    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245993    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 18 20:20:31.251817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245996    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.245999    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246002    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246004    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246007    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246024    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246027    2566 flags.go:64] FLAG: --feature-gates=""
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246031    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246034    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246037    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246041    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246044    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246047    2566 flags.go:64] FLAG: --help="false"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246050    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246053    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246056    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246059    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246062    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246065    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246068    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246072    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246075    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246078    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 20:20:31.252429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246081    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246085    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246088    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246090    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246093    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246097    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246100    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246103    2566 flags.go:64] FLAG: --lock-file=""
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246106    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246109    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246112    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246118    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246120    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246123    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246126    2566 flags.go:64] FLAG: --logging-format="text"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246129    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246133    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246136    2566 flags.go:64] FLAG: --manifest-url=""
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246138    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246143    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246146    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246149    2566 flags.go:64] FLAG: --max-pods="110"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246152    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246155    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246158    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 20:20:31.253048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246161    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246164    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246167    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246170    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246178    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246181    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246184    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246187    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246190    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246195    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246198    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246201    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246206    2566 flags.go:64] FLAG: --port="10250"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246209    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246211    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e5e43d9d56fdfa03"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246215    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246218    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246221    2566 flags.go:64] FLAG: --register-node="true"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246224    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246227    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246231    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246233    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246236    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246240    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246243    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 20:20:31.253671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246246    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246249    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246252    2566 flags.go:64] FLAG: --runonce="false"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246254    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246257    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246260    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246263    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246266    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246269    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246272    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246275    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246278    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246282    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246285    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246288    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246291    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246295    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246298    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246303    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246306    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246309    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246314    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246316    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246319    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246322    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 20:20:31.254306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246325    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246328    2566 flags.go:64] FLAG: --v="2"
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246331    2566 flags.go:64] FLAG: --version="false"
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246335    2566 flags.go:64] FLAG: --vmodule=""
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246339    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.246342    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246457    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246461    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246464    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246467    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246470    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246473    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246476    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246479    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246481    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246484    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246487    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246489    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246492    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246497    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246499    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:31.254936 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246503    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246505    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246508    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246512    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246516    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246519    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246522    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246525    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246527    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246530    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246533    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246535    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246538    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246541    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246543    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246546    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246548    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246551    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246554    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:31.255459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246558    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246562    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246565    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246568    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246571    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246573    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246576    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246578    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246581    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246584    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246586    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246590    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246593    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246596    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246599    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246602    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246604    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246607    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246610    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:31.255919 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246613    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246615    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246619    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246621    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246624    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246627    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246629    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246632    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246635    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246637    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246640    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246643    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246645    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246648    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246650    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246653    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246656    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246659    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246661    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246664    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:31.256493 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246666    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246669    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246672    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246674    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246678    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246681    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246683    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246686    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246689    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246692    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246695    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246698    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246700    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246703    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246706    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246709    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246713    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246716    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246719    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246722    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:31.256964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246724    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246727    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246731    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246733    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246736    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.246739    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.247604    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.254772    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.254785    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.255616    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.255624    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255692    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255698    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255701    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:31.257459 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255705    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255708    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255711    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255715    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255717    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255720    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255723    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255726    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255729    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255737    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255740    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255742    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255745    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255748    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255750    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255753    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255755    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255758    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255761    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255764    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:31.257809 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255766    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255769    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255771    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255774    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255778    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255781    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255784    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255788    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255792    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255795    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255798    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255801    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255804    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255806    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255809    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255812    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255814    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255817    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255819    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255822    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:31.258338 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255825    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255827    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255835    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255838    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255840    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255843    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255845    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255848    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255851    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255853    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255856    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255858    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255861    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255863    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255866    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255869    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255871    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255874    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255877    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255885    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:31.258886 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255888    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255890    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255893    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255895    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255898    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255901    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255903    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255906    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255908    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255911    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255913    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255916    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255919    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255923    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255926    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255936    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255939    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255941    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255944    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:31.259373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255947    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255949    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255952    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255956    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255959    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255962    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255965    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255968    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255971    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255973    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255976    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255980    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255982    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255985    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255987    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255990    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:31.259839 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.255992    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.255997    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256161    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256166    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256169    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256173    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256175    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256178    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256181    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256184    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256187    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256189    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256192    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256200    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 20:20:31.260270 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256202    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256205    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256207    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256210    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256212    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256215    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256218    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256220    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256223    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256225    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256228    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256230    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256233    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256236    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256238    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256241    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256244    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256247    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256250    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256252    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 20:20:31.260609 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256256    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256260    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256263    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256266    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256268    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256271    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256273    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256276    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256280    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256283    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256286    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256289    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256297    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256301    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256303    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256306    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256309    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256311    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256314    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256317    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 20:20:31.261104 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256319    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256322    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256324    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256327    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256329    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256332    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256334    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256337    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256339    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256342    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256345    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256347    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256350    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256353    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256357    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256360    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256363    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256365    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256368    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256370    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 20:20:31.261592 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256373    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256376    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256378    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256381    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256383    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256391    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256394    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256399    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256402    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256404    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256407    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256409    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256412    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256415    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256417    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256420    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256422    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256425    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256427    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256430    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 20:20:31.262187 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256433    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256436    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256438    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256441    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256443    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256446    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:31.256449    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.256454    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.256559    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.258508    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.259502    2566 server.go:1019] "Starting client certificate rotation"
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.259600    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 20:20:31.262917 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.259630    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 20:20:31.264084 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.264068    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 20:20:31.265224 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.265206    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 20:20:31.278961 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.278944    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 18 20:20:31.282623 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.282604    2566 log.go:25] "Validated CRI v1 image API"
Sep 18 20:20:31.284268 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.284251    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 20:20:31.288953 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.288931    2566 fs.go:135] Filesystem UUIDs: map[509c151b-2c6d-4c4d-ad82-d5f2dd7d5223:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a5f02a50-3bb9-45ae-8dca-9a101cc3dc3c:/dev/nvme0n1p3]
Sep 18 20:20:31.289031 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.288953    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 20:20:31.292364 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.292345    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 20:20:31.294367 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.294259    2566 manager.go:216] Machine: {Timestamp:2026-09-18 20:20:31.29297351 +0000 UTC m=+0.323899642 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092225 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec202dfeb2c975f5be3a5521d15877d9 SystemUUID:ec202dfe-b2c9-75f5-be3a-5521d15877d9 BootID:7121a7cd-47fd-4a82-8105-f5ae0a6484a2 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c3:d7:4d:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c3:d7:4d:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:4a:b1:d4:7f:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 20:20:31.294367 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.294364    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 20:20:31.294496 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.294470    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 20:20:31.297075 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.297052    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 20:20:31.297252 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.297077    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-76.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 20:20:31.297782 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.297770    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 20:20:31.297834 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.297790    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 20:20:31.297834 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.297801    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 20:20:31.298473 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.298461    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 20:20:31.298559 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.298549    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 20:20:31.301918 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.301905    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 20:20:31.302104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.302093    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 20:20:31.302164 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.302129    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 20:20:31.302164 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.302148    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 20:20:31.302259 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.302167    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 20:20:31.302259 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.302188    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 18 20:20:31.302259 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.302201    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 20:20:31.302601 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.302585    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 20:20:31.302671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.302614    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 20:20:31.304931 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.304910    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 20:20:31.307407 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.307391    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 20:20:31.307486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.307422    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 20:20:31.308903 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308891    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 20:20:31.308937 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308910    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 20:20:31.308937 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308917    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 20:20:31.308937 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308923    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 20:20:31.308937 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308931    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 20:20:31.308937 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308937    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 20:20:31.309084 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308950    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 20:20:31.309084 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308956    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 20:20:31.309084 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308963    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 20:20:31.309084 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308969    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 20:20:31.309084 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.308993    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 20:20:31.309084 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.309000    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 20:20:31.309590 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.309581    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 20:20:31.309620 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.309592    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 20:20:31.313148 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.313127    2566 server.go:1268] "Started kubelet"
Sep 18 20:20:31.313274 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.313248    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 20:20:31.313715 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.313258    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 20:20:31.313715 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.313334    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 20:20:31.313955 ip-10-0-143-76 systemd[1]: Started Kubernetes Kubelet.
Sep 18 20:20:31.314371 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.314354    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 20:20:31.315645 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.315632    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 20:20:31.315717 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.315684    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-76.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 20:20:31.315717 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.315677    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 20:20:31.315849 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.315824    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 20:20:31.320797 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.320776    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bg8t7"
Sep 18 20:20:31.321007 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.320028    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-76.ec2.internal.18d683d0009d5847  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-76.ec2.internal,UID:ip-10-0-143-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-76.ec2.internal,},FirstTimestamp:2026-09-18 20:20:31.313098823 +0000 UTC m=+0.344024953,LastTimestamp:2026-09-18 20:20:31.313098823 +0000 UTC m=+0.344024953,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-76.ec2.internal,}"
Sep 18 20:20:31.321156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.321123    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 20:20:31.321156 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.321118    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 20:20:31.321423 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.321409    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 20:20:31.321636 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.321617    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:31.321907 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.321887    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 20:20:31.321907 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.321889    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 20:20:31.322050 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.321917    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 20:20:31.322050 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.322042    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 20:20:31.322137 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.322056    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 20:20:31.322533 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.322517    2566 factory.go:153] Registering CRI-O factory
Sep 18 20:20:31.322533 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.322532    2566 factory.go:223] Registration of the crio container factory successfully
Sep 18 20:20:31.322654 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.322588    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 20:20:31.322654 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.322609    2566 factory.go:55] Registering systemd factory
Sep 18 20:20:31.322654 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.322615    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 18 20:20:31.322654 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.322631    2566 factory.go:103] Registering Raw factory
Sep 18 20:20:31.322654 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.322643    2566 manager.go:1196] Started watching for new ooms in manager
Sep 18 20:20:31.323040 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.323028    2566 manager.go:349] Starting recovery of all containers
Sep 18 20:20:31.327211 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.327184    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bg8t7"
Sep 18 20:20:31.327293 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.327256    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 20:20:31.327362 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.327342    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 20:20:31.334507 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.334485    2566 manager.go:354] Recovery completed
Sep 18 20:20:31.339100 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.339085    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:31.340918 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.340896    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:31.340983 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.340932    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:31.340983 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.340946    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:31.341529 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.341504    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 20:20:31.341529 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.341517    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 20:20:31.341529 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.341532    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 20:20:31.341697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.341562    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:31.342288 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.342269    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:31.342352 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.342300    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:31.342352 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.342312    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:31.342952 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.342886    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-76.ec2.internal.18d683d00245d0c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-76.ec2.internal,UID:ip-10-0-143-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-76.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-76.ec2.internal,},FirstTimestamp:2026-09-18 20:20:31.340916936 +0000 UTC m=+0.371843070,LastTimestamp:2026-09-18 20:20:31.340916936 +0000 UTC m=+0.371843070,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-76.ec2.internal,}"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.422196    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.452957    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.454242    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.454291    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.454316    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.454354    2566 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.456751    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.488694    2566 policy_none.go:49] "None policy: Start"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.488718    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 20:20:31.489753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.488730    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 20:20:31.490779 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.490766    2566 policy_none.go:47] "Start"
Sep 18 20:20:31.522419 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.522394    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:31.522508 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.522496    2566 manager.go:337] "Starting Device Plugin manager"
Sep 18 20:20:31.522559 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.522530    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 20:20:31.522559 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.522540    2566 server.go:87] "Starting device plugin registration server"
Sep 18 20:20:31.522759 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.522749    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 20:20:31.522801 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.522760    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 20:20:31.522883 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.522869    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 20:20:31.522954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.522943    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 20:20:31.522954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.522954    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 20:20:31.523495 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.523478    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 20:20:31.523561 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.523523    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:31.532618 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.532602    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-76.ec2.internal\" not found" node="ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.554713 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.554686    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal"]
Sep 18 20:20:31.554794 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.554785    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:31.556245 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.556229    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:31.556329 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.556262    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:31.556329 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.556278    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:31.558717 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.558702    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:31.558830 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.558810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.558875 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.558836    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:31.559416 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.559396    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:31.559558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.559423    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:31.559558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.559438    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:31.559558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.559501    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:31.559558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.559524    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:31.559558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.559533    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:31.561688 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.561668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.561756 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.561723    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:31.562402 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.562383    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:31.562495 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.562413    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:31.562495 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.562426    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:31.577529 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.577509    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-76.ec2.internal\" not found" node="ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.581315 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.581300    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-76.ec2.internal\" not found" node="ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.623219 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.623202    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 20:20:31.623312 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.623251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ed2e113751e9f2a9cf8de7dcd23788f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal\" (UID: \"2ed2e113751e9f2a9cf8de7dcd23788f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.623312 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.623285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed2e113751e9f2a9cf8de7dcd23788f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal\" (UID: \"2ed2e113751e9f2a9cf8de7dcd23788f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.623312 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.623308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36964665b9e807b88bd927b05ac588ed-config\") pod \"kube-apiserver-proxy-ip-10-0-143-76.ec2.internal\" (UID: \"36964665b9e807b88bd927b05ac588ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.623986 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.623971    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 20:20:31.624070 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.623996    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 20:20:31.624070 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.624007    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeHasSufficientPID"
Sep 18 20:20:31.624070 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.624052    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.629782 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.629768    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.629847 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.629787    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-76.ec2.internal\": node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:31.644415 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.644396    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:31.723568 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.723539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ed2e113751e9f2a9cf8de7dcd23788f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal\" (UID: \"2ed2e113751e9f2a9cf8de7dcd23788f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.723568 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.723569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed2e113751e9f2a9cf8de7dcd23788f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal\" (UID: \"2ed2e113751e9f2a9cf8de7dcd23788f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.723704 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.723585    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36964665b9e807b88bd927b05ac588ed-config\") pod \"kube-apiserver-proxy-ip-10-0-143-76.ec2.internal\" (UID: \"36964665b9e807b88bd927b05ac588ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.723704 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.723615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36964665b9e807b88bd927b05ac588ed-config\") pod \"kube-apiserver-proxy-ip-10-0-143-76.ec2.internal\" (UID: \"36964665b9e807b88bd927b05ac588ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.723704 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.723633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ed2e113751e9f2a9cf8de7dcd23788f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal\" (UID: \"2ed2e113751e9f2a9cf8de7dcd23788f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.723704 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.723640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed2e113751e9f2a9cf8de7dcd23788f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal\" (UID: \"2ed2e113751e9f2a9cf8de7dcd23788f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.744692 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.744639    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:31.845366 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.845340    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:31.880518 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.880494    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.884546 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:31.884524    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:31.945857 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:31.945829    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:32.046404 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:32.046324    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:32.147745 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:32.147715    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:32.249575 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:32.249547    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:32.259818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.259793    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 20:20:32.259958 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.259941    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 20:20:32.333368 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.333297    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 20:15:31 +0000 UTC" deadline="2028-02-22 13:39:08.202040292 +0000 UTC"
Sep 18 20:20:32.333368 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.333324    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12521h18m35.868719081s"
Sep 18 20:20:32.350160 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:32.349912    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:32.350160 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.349987    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:32.451181 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:32.451148    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:32.551490 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:32.551467    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:32.560826 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:32.560792    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36964665b9e807b88bd927b05ac588ed.slice/crio-c87865597d9b55b72cc8668f7d55e1876d10925ee0d790e44a52edc25a514061 WatchSource:0}: Error finding container c87865597d9b55b72cc8668f7d55e1876d10925ee0d790e44a52edc25a514061: Status 404 returned error can't find the container with id c87865597d9b55b72cc8668f7d55e1876d10925ee0d790e44a52edc25a514061
Sep 18 20:20:32.561762 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:32.561728    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed2e113751e9f2a9cf8de7dcd23788f.slice/crio-0c3de94037f02a402070db38f6a7e976f397af21ce26896051dd2d50309004cb WatchSource:0}: Error finding container 0c3de94037f02a402070db38f6a7e976f397af21ce26896051dd2d50309004cb: Status 404 returned error can't find the container with id 0c3de94037f02a402070db38f6a7e976f397af21ce26896051dd2d50309004cb
Sep 18 20:20:32.568615 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.568601    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:20:32.653051 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:32.652940    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-76.ec2.internal\" not found"
Sep 18 20:20:32.695406 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.695382    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:32.722112 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.722079    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:32.732596 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.732567    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:32.735598 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.735581    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 20:20:32.736494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.736480    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal"
Sep 18 20:20:32.745117 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:32.745100    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 20:20:33.305999 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.305958    2566 apiserver.go:52] "Watching apiserver"
Sep 18 20:20:33.318715 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.317794    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 20:20:33.318715 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.318298    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal","openshift-network-diagnostics/network-check-target-75k5k","openshift-ovn-kubernetes/ovnkube-node-7xnhp","kube-system/global-pull-secret-syncer-msj7n","kube-system/konnectivity-agent-mh7w7","kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal","openshift-image-registry/node-ca-z7w45","openshift-multus/multus-2l4jx","openshift-multus/multus-additional-cni-plugins-k8m8l","openshift-multus/network-metrics-daemon-zfhl9","openshift-network-operator/iptables-alerter-b7b5n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c","openshift-cluster-node-tuning-operator/tuned-vzx2c","openshift-dns/node-resolver-hglx4"]
Sep 18 20:20:33.321177 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.321145    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.322464 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.321737    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 20:20:33.323965 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.323945    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.325227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.325205    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b2pt7\""
Sep 18 20:20:33.325549 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.325531    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 20:20:33.325750 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.325733    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 20:20:33.326861 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.326839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.327621 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.327441    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:20:33.327621 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.327495    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-blnlc\""
Sep 18 20:20:33.327621 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.327538    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 20:20:33.327621 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.327478    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 20:20:33.328785 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.328761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:33.328897 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.328832    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:33.329954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.329935    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 20:20:33.330492 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.330473    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 20:20:33.330579 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.330493    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 20:20:33.330795 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.330777    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 20:20:33.330952 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.330935    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 20:20:33.331022 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.331004    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 20:20:33.331287 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.331269    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x4hql\""
Sep 18 20:20:33.331455 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.331438    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:33.332412 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-systemd\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.332520 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332433    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-run\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.332520 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96v4v\" (UniqueName: \"kubernetes.io/projected/7def255f-b389-4260-b3bd-cb1ce422bef6-kube-api-access-96v4v\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.332520 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbpq\" (UniqueName: \"kubernetes.io/projected/d8e15a8a-8986-42a3-9cba-945c5b6894a4-kube-api-access-slbpq\") pod \"node-ca-z7w45\" (UID: \"d8e15a8a-8986-42a3-9cba-945c5b6894a4\") " pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.332692 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-modprobe-d\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.332692 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332546    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-kubernetes\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.332692 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-var-lib-kubelet\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.332692 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-tuned\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.332692 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332619    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-sys\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.332692 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8e15a8a-8986-42a3-9cba-945c5b6894a4-serviceca\") pod \"node-ca-z7w45\" (UID: \"d8e15a8a-8986-42a3-9cba-945c5b6894a4\") " pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.332692 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-sysctl-d\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.333026 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-sysctl-conf\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.333026 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-host\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.333026 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-sysconfig\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.333026 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-lib-modules\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.333026 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7def255f-b389-4260-b3bd-cb1ce422bef6-tmp\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.333026 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.332817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8e15a8a-8986-42a3-9cba-945c5b6894a4-host\") pod \"node-ca-z7w45\" (UID: \"d8e15a8a-8986-42a3-9cba-945c5b6894a4\") " pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.333872 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.333852    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 20:20:33.334029 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.333885    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.334262 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.334241    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4v68c\""
Sep 18 20:20:33.334393 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.334347    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 20:20:33.334459 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.334409    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 20:20:33.336437 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.336059    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.336437 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.336304    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ctj5q\""
Sep 18 20:20:33.337752 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.336588    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 20:20:33.337752 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.337024    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 20:20:33.339344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.338388    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.339344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.339248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 20:20:33.343465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.339621    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 20:20:33.343465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.339661    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 20:20:33.343465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.339675    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bdqq7\""
Sep 18 20:20:33.343465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.339951    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 20:20:33.343465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.340002    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 20:20:33.343465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.341050    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:33.343465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.341052    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zz9tb\""
Sep 18 20:20:33.343465 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.341115    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:33.343465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.341335    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 20:20:33.344063 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.343540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:33.344063 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.343614    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:33.347072 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.346122    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.348618 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.348518    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 20:20:33.348618 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.348587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.349011 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.348862    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 20:20:33.349298 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.349278    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f4zkm\""
Sep 18 20:20:33.349450 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.349432    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 20:20:33.351186 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.351094    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 20:20:33.351511 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.351482    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 20:20:33.351664 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.351640    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lqjpx\""
Sep 18 20:20:33.352066 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.351889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 20:20:33.359461 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.359428    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rj6nm"
Sep 18 20:20:33.369162 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.369137    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rj6nm"
Sep 18 20:20:33.423315 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.423289    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 20:20:33.433374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-sys\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.433494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:33.433494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-kubelet\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.433494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-slash\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.433494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce0abf55-e019-4d4b-8bab-45921f3645c2-ovnkube-config\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.433494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c273cf8-0881-435f-be53-928e52dd5a58-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.433494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433475    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-sys\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.433814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-conf-dir\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.433814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-run-openvswitch\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.433814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.433814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c273cf8-0881-435f-be53-928e52dd5a58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.433814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-daemon-config\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.433814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42jg\" (UniqueName: \"kubernetes.io/projected/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-kube-api-access-q42jg\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.433814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96v4v\" (UniqueName: \"kubernetes.io/projected/7def255f-b389-4260-b3bd-cb1ce422bef6-kube-api-access-96v4v\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.433814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jqhq\" (UniqueName: \"kubernetes.io/projected/d5396b35-d934-4cbd-bf7a-60e0ecb853f7-kube-api-access-9jqhq\") pod \"node-resolver-hglx4\" (UID: \"d5396b35-d934-4cbd-bf7a-60e0ecb853f7\") " pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-run-systemd\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbpq\" (UniqueName: \"kubernetes.io/projected/d8e15a8a-8986-42a3-9cba-945c5b6894a4-kube-api-access-slbpq\") pod \"node-ca-z7w45\" (UID: \"d8e15a8a-8986-42a3-9cba-945c5b6894a4\") " pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.433993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-socket-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-sys-fs\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-var-lib-cni-multus\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c273cf8-0881-435f-be53-928e52dd5a58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-cni-dir\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-hostroot\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-lib-modules\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7def255f-b389-4260-b3bd-cb1ce422bef6-tmp\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-run-ovn\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-cni-netd\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce0abf55-e019-4d4b-8bab-45921f3645c2-env-overrides\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-run-k8s-cni-cncf-io\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-etc-kubernetes\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.434346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8e15a8a-8986-42a3-9cba-945c5b6894a4-host\") pod \"node-ca-z7w45\" (UID: \"d8e15a8a-8986-42a3-9cba-945c5b6894a4\") " pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8e15a8a-8986-42a3-9cba-945c5b6894a4-host\") pod \"node-ca-z7w45\" (UID: \"d8e15a8a-8986-42a3-9cba-945c5b6894a4\") " pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-lib-modules\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-device-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-run-netns\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c273cf8-0881-435f-be53-928e52dd5a58-cnibin\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkd9\" (UniqueName: \"kubernetes.io/projected/4e160c7e-4ea8-48e5-b932-32be44d1c289-kube-api-access-vqkd9\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-kubernetes\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a406b73-2c97-49f0-82b9-964afe466106-dbus\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5396b35-d934-4cbd-bf7a-60e0ecb853f7-tmp-dir\") pod \"node-resolver-hglx4\" (UID: \"d5396b35-d934-4cbd-bf7a-60e0ecb853f7\") " pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434663    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb8b8b3-f9f1-41eb-b7d0-d571159c441a-iptables-alerter-script\") pod \"iptables-alerter-b7b5n\" (UID: \"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a\") " pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-systemd-units\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-kubernetes\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-log-socket\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/325de808-8e01-460b-9507-866c7181e24d-agent-certs\") pod \"konnectivity-agent-mh7w7\" (UID: \"325de808-8e01-460b-9507-866c7181e24d\") " pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8e15a8a-8986-42a3-9cba-945c5b6894a4-serviceca\") pod \"node-ca-z7w45\" (UID: \"d8e15a8a-8986-42a3-9cba-945c5b6894a4\") " pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.435156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-sysctl-d\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-host\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxcm\" (UniqueName: \"kubernetes.io/projected/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-kube-api-access-xcxcm\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zq8\" (UniqueName: \"kubernetes.io/projected/ce0abf55-e019-4d4b-8bab-45921f3645c2-kube-api-access-g8zq8\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/325de808-8e01-460b-9507-866c7181e24d-konnectivity-ca\") pod \"konnectivity-agent-mh7w7\" (UID: \"325de808-8e01-460b-9507-866c7181e24d\") " pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-host\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434966    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-sysctl-d\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.434989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-sysconfig\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce0abf55-e019-4d4b-8bab-45921f3645c2-ovn-node-metrics-cert\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-sysconfig\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c273cf8-0881-435f-be53-928e52dd5a58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjzk7\" (UniqueName: \"kubernetes.io/projected/2c273cf8-0881-435f-be53-928e52dd5a58-kube-api-access-hjzk7\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-cnibin\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-systemd\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8e15a8a-8986-42a3-9cba-945c5b6894a4-serviceca\") pod \"node-ca-z7w45\" (UID: \"d8e15a8a-8986-42a3-9cba-945c5b6894a4\") " pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.435895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-run-multus-certs\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-modprobe-d\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-systemd\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5396b35-d934-4cbd-bf7a-60e0ecb853f7-hosts-file\") pod \"node-resolver-hglx4\" (UID: \"d5396b35-d934-4cbd-bf7a-60e0ecb853f7\") " pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce0abf55-e019-4d4b-8bab-45921f3645c2-ovnkube-script-lib\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-sysctl-conf\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-modprobe-d\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f86p\" (UniqueName: \"kubernetes.io/projected/7cb8b8b3-f9f1-41eb-b7d0-d571159c441a-kube-api-access-8f86p\") pod \"iptables-alerter-b7b5n\" (UID: \"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a\") " pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-etc-openvswitch\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435533    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-cni-binary-copy\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435555    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-sysctl-conf\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c273cf8-0881-435f-be53-928e52dd5a58-system-cni-dir\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-system-cni-dir\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-run-netns\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-run\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.436660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb8b8b3-f9f1-41eb-b7d0-d571159c441a-host-slash\") pod \"iptables-alerter-b7b5n\" (UID: \"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a\") " pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-registration-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-run\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-var-lib-kubelet\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-var-lib-kubelet\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-tuned\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a406b73-2c97-49f0-82b9-964afe466106-kubelet-config\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-var-lib-openvswitch\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-node-log\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-cni-bin\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435913    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-socket-dir-parent\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.435928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-var-lib-cni-bin\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.437674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.436102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7def255f-b389-4260-b3bd-cb1ce422bef6-var-lib-kubelet\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.439189 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.439165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7def255f-b389-4260-b3bd-cb1ce422bef6-tmp\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.439301 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.439165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7def255f-b389-4260-b3bd-cb1ce422bef6-etc-tuned\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.444779 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.444722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96v4v\" (UniqueName: \"kubernetes.io/projected/7def255f-b389-4260-b3bd-cb1ce422bef6-kube-api-access-96v4v\") pod \"tuned-vzx2c\" (UID: \"7def255f-b389-4260-b3bd-cb1ce422bef6\") " pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.446170 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.446146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbpq\" (UniqueName: \"kubernetes.io/projected/d8e15a8a-8986-42a3-9cba-945c5b6894a4-kube-api-access-slbpq\") pod \"node-ca-z7w45\" (UID: \"d8e15a8a-8986-42a3-9cba-945c5b6894a4\") " pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.459693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.459639    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal" event={"ID":"36964665b9e807b88bd927b05ac588ed","Type":"ContainerStarted","Data":"c87865597d9b55b72cc8668f7d55e1876d10925ee0d790e44a52edc25a514061"}
Sep 18 20:20:33.461178 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.461149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal" event={"ID":"2ed2e113751e9f2a9cf8de7dcd23788f","Type":"ContainerStarted","Data":"0c3de94037f02a402070db38f6a7e976f397af21ce26896051dd2d50309004cb"}
Sep 18 20:20:33.536708 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb8b8b3-f9f1-41eb-b7d0-d571159c441a-host-slash\") pod \"iptables-alerter-b7b5n\" (UID: \"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a\") " pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.536708 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-registration-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.536708 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-var-lib-kubelet\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-var-lib-kubelet\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb8b8b3-f9f1-41eb-b7d0-d571159c441a-host-slash\") pod \"iptables-alerter-b7b5n\" (UID: \"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a\") " pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536771    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-registration-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a406b73-2c97-49f0-82b9-964afe466106-kubelet-config\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-var-lib-openvswitch\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-node-log\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-cni-bin\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a406b73-2c97-49f0-82b9-964afe466106-kubelet-config\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.536975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-socket-dir-parent\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.537027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-socket-dir-parent\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-node-log\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-var-lib-cni-bin\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-cni-bin\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-var-lib-openvswitch\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537124    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-var-lib-cni-bin\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-kubelet\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-slash\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.537222    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce0abf55-e019-4d4b-8bab-45921f3645c2-ovnkube-config\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-kubelet\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c273cf8-0881-435f-be53-928e52dd5a58-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-slash\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.537295    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret podName:1a406b73-2c97-49f0-82b9-964afe466106 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:34.037279362 +0000 UTC m=+3.068205497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret") pod "global-pull-secret-syncer-msj7n" (UID: "1a406b73-2c97-49f0-82b9-964afe466106") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-conf-dir\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-run-openvswitch\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.537562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c273cf8-0881-435f-be53-928e52dd5a58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-daemon-config\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-conf-dir\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42jg\" (UniqueName: \"kubernetes.io/projected/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-kube-api-access-q42jg\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-run-openvswitch\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jqhq\" (UniqueName: \"kubernetes.io/projected/d5396b35-d934-4cbd-bf7a-60e0ecb853f7-kube-api-access-9jqhq\") pod \"node-resolver-hglx4\" (UID: \"d5396b35-d934-4cbd-bf7a-60e0ecb853f7\") " pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-run-systemd\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-socket-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-sys-fs\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-var-lib-cni-multus\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c273cf8-0881-435f-be53-928e52dd5a58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-cni-dir\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-hostroot\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-run-ovn\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.538595 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-cni-netd\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce0abf55-e019-4d4b-8bab-45921f3645c2-env-overrides\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-run-k8s-cni-cncf-io\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-etc-kubernetes\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-device-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce0abf55-e019-4d4b-8bab-45921f3645c2-ovnkube-config\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-run-netns\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.537995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c273cf8-0881-435f-be53-928e52dd5a58-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c273cf8-0881-435f-be53-928e52dd5a58-cnibin\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkd9\" (UniqueName: \"kubernetes.io/projected/4e160c7e-4ea8-48e5-b932-32be44d1c289-kube-api-access-vqkd9\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538046    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-daemon-config\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c273cf8-0881-435f-be53-928e52dd5a58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a406b73-2c97-49f0-82b9-964afe466106-dbus\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-multus-cni-dir\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-socket-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538154    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-run-ovn\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538175    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a406b73-2c97-49f0-82b9-964afe466106-dbus\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:33.539360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5396b35-d934-4cbd-bf7a-60e0ecb853f7-tmp-dir\") pod \"node-resolver-hglx4\" (UID: \"d5396b35-d934-4cbd-bf7a-60e0ecb853f7\") " pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538195    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-run-systemd\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb8b8b3-f9f1-41eb-b7d0-d571159c441a-iptables-alerter-script\") pod \"iptables-alerter-b7b5n\" (UID: \"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a\") " pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-systemd-units\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538248    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c273cf8-0881-435f-be53-928e52dd5a58-cnibin\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-etc-kubernetes\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-hostroot\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-var-lib-cni-multus\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-device-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-cni-netd\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-log-socket\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/325de808-8e01-460b-9507-866c7181e24d-agent-certs\") pod \"konnectivity-agent-mh7w7\" (UID: \"325de808-8e01-460b-9507-866c7181e24d\") " pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxcm\" (UniqueName: \"kubernetes.io/projected/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-kube-api-access-xcxcm\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538544    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c273cf8-0881-435f-be53-928e52dd5a58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zq8\" (UniqueName: \"kubernetes.io/projected/ce0abf55-e019-4d4b-8bab-45921f3645c2-kube-api-access-g8zq8\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5396b35-d934-4cbd-bf7a-60e0ecb853f7-tmp-dir\") pod \"node-resolver-hglx4\" (UID: \"d5396b35-d934-4cbd-bf7a-60e0ecb853f7\") " pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.540314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/325de808-8e01-460b-9507-866c7181e24d-konnectivity-ca\") pod \"konnectivity-agent-mh7w7\" (UID: \"325de808-8e01-460b-9507-866c7181e24d\") " pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-run-k8s-cni-cncf-io\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-run-netns\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce0abf55-e019-4d4b-8bab-45921f3645c2-ovn-node-metrics-cert\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c273cf8-0881-435f-be53-928e52dd5a58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-sys-fs\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-systemd-units\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.539958    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/325de808-8e01-460b-9507-866c7181e24d-konnectivity-ca\") pod \"konnectivity-agent-mh7w7\" (UID: \"325de808-8e01-460b-9507-866c7181e24d\") " pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.540421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c273cf8-0881-435f-be53-928e52dd5a58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce0abf55-e019-4d4b-8bab-45921f3645c2-env-overrides\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.538571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-log-socket\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjzk7\" (UniqueName: \"kubernetes.io/projected/2c273cf8-0881-435f-be53-928e52dd5a58-kube-api-access-hjzk7\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-cnibin\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541154    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb8b8b3-f9f1-41eb-b7d0-d571159c441a-iptables-alerter-script\") pod \"iptables-alerter-b7b5n\" (UID: \"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a\") " pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:33.541194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-run-multus-certs\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5396b35-d934-4cbd-bf7a-60e0ecb853f7-hosts-file\") pod \"node-resolver-hglx4\" (UID: \"d5396b35-d934-4cbd-bf7a-60e0ecb853f7\") " pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce0abf55-e019-4d4b-8bab-45921f3645c2-ovnkube-script-lib\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f86p\" (UniqueName: \"kubernetes.io/projected/7cb8b8b3-f9f1-41eb-b7d0-d571159c441a-kube-api-access-8f86p\") pod \"iptables-alerter-b7b5n\" (UID: \"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a\") " pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-etc-openvswitch\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-cni-binary-copy\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c273cf8-0881-435f-be53-928e52dd5a58-system-cni-dir\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-system-cni-dir\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-run-netns\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-run-netns\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.541889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-cnibin\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.542398 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.541996    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-host-run-multus-certs\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.542398 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.542075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5396b35-d934-4cbd-bf7a-60e0ecb853f7-hosts-file\") pod \"node-resolver-hglx4\" (UID: \"d5396b35-d934-4cbd-bf7a-60e0ecb853f7\") " pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.542915 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.542891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce0abf55-e019-4d4b-8bab-45921f3645c2-ovnkube-script-lib\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.543034 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.543014    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:33.543354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.543324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-cni-binary-copy\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.543480 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.543462    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs podName:4e160c7e-4ea8-48e5-b932-32be44d1c289 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:34.043440463 +0000 UTC m=+3.074366599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs") pod "network-metrics-daemon-zfhl9" (UID: "4e160c7e-4ea8-48e5-b932-32be44d1c289") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:33.543647 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.543604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-etc-openvswitch\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.543746 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.543725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c273cf8-0881-435f-be53-928e52dd5a58-system-cni-dir\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.543746 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.543749    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce0abf55-e019-4d4b-8bab-45921f3645c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.543886 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.543811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-system-cni-dir\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.547746 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.547692    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:33.547746 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.547727    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:33.547945 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.547755    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v2ll7 for pod openshift-network-diagnostics/network-check-target-75k5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:33.548362 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:33.548342    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7 podName:01e6577b-6859-4947-9532-fc0b9edcf3cb nodeName:}" failed. No retries permitted until 2026-09-18 20:20:34.048299719 +0000 UTC m=+3.079226087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v2ll7" (UniqueName: "kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7") pod "network-check-target-75k5k" (UID: "01e6577b-6859-4947-9532-fc0b9edcf3cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:33.549061 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.549033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42jg\" (UniqueName: \"kubernetes.io/projected/b2f71b87-9bf3-454d-a2e1-97a0ffa97f52-kube-api-access-q42jg\") pod \"multus-2l4jx\" (UID: \"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52\") " pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.549567 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.549469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jqhq\" (UniqueName: \"kubernetes.io/projected/d5396b35-d934-4cbd-bf7a-60e0ecb853f7-kube-api-access-9jqhq\") pod \"node-resolver-hglx4\" (UID: \"d5396b35-d934-4cbd-bf7a-60e0ecb853f7\") " pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.551207 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.551163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zq8\" (UniqueName: \"kubernetes.io/projected/ce0abf55-e019-4d4b-8bab-45921f3645c2-kube-api-access-g8zq8\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.551307 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.551289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/325de808-8e01-460b-9507-866c7181e24d-agent-certs\") pod \"konnectivity-agent-mh7w7\" (UID: \"325de808-8e01-460b-9507-866c7181e24d\") " pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:33.552285 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.551721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjzk7\" (UniqueName: \"kubernetes.io/projected/2c273cf8-0881-435f-be53-928e52dd5a58-kube-api-access-hjzk7\") pod \"multus-additional-cni-plugins-k8m8l\" (UID: \"2c273cf8-0881-435f-be53-928e52dd5a58\") " pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.552285 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.551930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkd9\" (UniqueName: \"kubernetes.io/projected/4e160c7e-4ea8-48e5-b932-32be44d1c289-kube-api-access-vqkd9\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:33.552285 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.552144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce0abf55-e019-4d4b-8bab-45921f3645c2-ovn-node-metrics-cert\") pod \"ovnkube-node-7xnhp\" (UID: \"ce0abf55-e019-4d4b-8bab-45921f3645c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.554404 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.554364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxcm\" (UniqueName: \"kubernetes.io/projected/dc154dc3-77eb-4bf4-a48a-daad8c7385bf-kube-api-access-xcxcm\") pod \"aws-ebs-csi-driver-node-dz75c\" (UID: \"dc154dc3-77eb-4bf4-a48a-daad8c7385bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:33.554581 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.554552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f86p\" (UniqueName: \"kubernetes.io/projected/7cb8b8b3-f9f1-41eb-b7d0-d571159c441a-kube-api-access-8f86p\") pod \"iptables-alerter-b7b5n\" (UID: \"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a\") " pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.595522 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.595439    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 20:20:33.637860 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.637823    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z7w45"
Sep 18 20:20:33.650308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.650275    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vzx2c"
Sep 18 20:20:33.660382 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.660352    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:33.673452 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.673420    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:33.675422 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.675401    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hglx4"
Sep 18 20:20:33.684842 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.684824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8m8l"
Sep 18 20:20:33.692573 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.692545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2l4jx"
Sep 18 20:20:33.705383 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.705359    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b7b5n"
Sep 18 20:20:33.712208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:33.712190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c"
Sep 18 20:20:34.046399 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:34.046296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:34.046399 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:34.046362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:34.046627 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:34.046461    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:34.046627 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:34.046511    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:34.046627 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:34.046551    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs podName:4e160c7e-4ea8-48e5-b932-32be44d1c289 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:35.046527578 +0000 UTC m=+4.077453754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs") pod "network-metrics-daemon-zfhl9" (UID: "4e160c7e-4ea8-48e5-b932-32be44d1c289") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:34.046627 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:34.046577    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret podName:1a406b73-2c97-49f0-82b9-964afe466106 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:35.046565506 +0000 UTC m=+4.077491630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret") pod "global-pull-secret-syncer-msj7n" (UID: "1a406b73-2c97-49f0-82b9-964afe466106") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:34.147122 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:34.147084    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:34.147304 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:34.147277    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:34.147304 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:34.147299    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:34.147416 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:34.147310    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v2ll7 for pod openshift-network-diagnostics/network-check-target-75k5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:34.147416 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:34.147367    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7 podName:01e6577b-6859-4947-9532-fc0b9edcf3cb nodeName:}" failed. No retries permitted until 2026-09-18 20:20:35.147354127 +0000 UTC m=+4.178280251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2ll7" (UniqueName: "kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7") pod "network-check-target-75k5k" (UID: "01e6577b-6859-4947-9532-fc0b9edcf3cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:34.373153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:34.373064    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 20:15:33 +0000 UTC" deadline="2028-04-21 14:58:27.296360612 +0000 UTC"
Sep 18 20:20:34.373153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:34.373097    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13938h37m52.923267121s"
Sep 18 20:20:34.619924 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:34.619854    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc154dc3_77eb_4bf4_a48a_daad8c7385bf.slice/crio-ec6bbdc2bb5ec884473fc4b83767692b4906550466d3cd6b163324048b36130c WatchSource:0}: Error finding container ec6bbdc2bb5ec884473fc4b83767692b4906550466d3cd6b163324048b36130c: Status 404 returned error can't find the container with id ec6bbdc2bb5ec884473fc4b83767692b4906550466d3cd6b163324048b36130c
Sep 18 20:20:34.622556 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:34.622239    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c273cf8_0881_435f_be53_928e52dd5a58.slice/crio-0c198bac66a7946af9ff017d89b6d63f9cf9da492c884675c67afcff48131e7b WatchSource:0}: Error finding container 0c198bac66a7946af9ff017d89b6d63f9cf9da492c884675c67afcff48131e7b: Status 404 returned error can't find the container with id 0c198bac66a7946af9ff017d89b6d63f9cf9da492c884675c67afcff48131e7b
Sep 18 20:20:34.628999 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:34.628978    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325de808_8e01_460b_9507_866c7181e24d.slice/crio-1efb9ddb343ab69cdc88db7f1ff2fd319ba083d42ae17829c912c9c887250aa5 WatchSource:0}: Error finding container 1efb9ddb343ab69cdc88db7f1ff2fd319ba083d42ae17829c912c9c887250aa5: Status 404 returned error can't find the container with id 1efb9ddb343ab69cdc88db7f1ff2fd319ba083d42ae17829c912c9c887250aa5
Sep 18 20:20:34.650511 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:34.650487    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7def255f_b389_4260_b3bd_cb1ce422bef6.slice/crio-d6200ce187ee46e0df56df69418ea256e9d6e54f01fff33eb6beb6a24d14890c WatchSource:0}: Error finding container d6200ce187ee46e0df56df69418ea256e9d6e54f01fff33eb6beb6a24d14890c: Status 404 returned error can't find the container with id d6200ce187ee46e0df56df69418ea256e9d6e54f01fff33eb6beb6a24d14890c
Sep 18 20:20:34.651182 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:34.651158    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5396b35_d934_4cbd_bf7a_60e0ecb853f7.slice/crio-ac985710eecfaf329fe17de133c97dbee54b7460b81cc70cc7be1dc7ed03c943 WatchSource:0}: Error finding container ac985710eecfaf329fe17de133c97dbee54b7460b81cc70cc7be1dc7ed03c943: Status 404 returned error can't find the container with id ac985710eecfaf329fe17de133c97dbee54b7460b81cc70cc7be1dc7ed03c943
Sep 18 20:20:34.652175 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:34.652147    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f71b87_9bf3_454d_a2e1_97a0ffa97f52.slice/crio-5ac859be51155e45dcdc5759c2dff6188db19fc210d365bd32f1d9e05544e586 WatchSource:0}: Error finding container 5ac859be51155e45dcdc5759c2dff6188db19fc210d365bd32f1d9e05544e586: Status 404 returned error can't find the container with id 5ac859be51155e45dcdc5759c2dff6188db19fc210d365bd32f1d9e05544e586
Sep 18 20:20:34.653073 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:34.653043    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0abf55_e019_4d4b_8bab_45921f3645c2.slice/crio-ea7749f2e0cf4a62a6bd02e7ac92e94ce380cd70c8212ed4a30936e69382cc0b WatchSource:0}: Error finding container ea7749f2e0cf4a62a6bd02e7ac92e94ce380cd70c8212ed4a30936e69382cc0b: Status 404 returned error can't find the container with id ea7749f2e0cf4a62a6bd02e7ac92e94ce380cd70c8212ed4a30936e69382cc0b
Sep 18 20:20:34.654005 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:34.653901    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e15a8a_8986_42a3_9cba_945c5b6894a4.slice/crio-8e20fa996d871c28f82a4f0cebc29597bcccb60640dddb44a625552538533f55 WatchSource:0}: Error finding container 8e20fa996d871c28f82a4f0cebc29597bcccb60640dddb44a625552538533f55: Status 404 returned error can't find the container with id 8e20fa996d871c28f82a4f0cebc29597bcccb60640dddb44a625552538533f55
Sep 18 20:20:34.654914 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:20:34.654892    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb8b8b3_f9f1_41eb_b7d0_d571159c441a.slice/crio-5be58ceab42b0910c670cfaf32730459c6596b338380251ae6d174f7dc2b3e60 WatchSource:0}: Error finding container 5be58ceab42b0910c670cfaf32730459c6596b338380251ae6d174f7dc2b3e60: Status 404 returned error can't find the container with id 5be58ceab42b0910c670cfaf32730459c6596b338380251ae6d174f7dc2b3e60
Sep 18 20:20:35.055259 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.054715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:35.055259 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.054845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:35.055259 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.054872    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:35.055259 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.054950    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:35.055259 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.054951    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret podName:1a406b73-2c97-49f0-82b9-964afe466106 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:37.054931458 +0000 UTC m=+6.085857590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret") pod "global-pull-secret-syncer-msj7n" (UID: "1a406b73-2c97-49f0-82b9-964afe466106") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:35.055259 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.055014    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs podName:4e160c7e-4ea8-48e5-b932-32be44d1c289 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:37.054998023 +0000 UTC m=+6.085924153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs") pod "network-metrics-daemon-zfhl9" (UID: "4e160c7e-4ea8-48e5-b932-32be44d1c289") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:35.156332 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.155587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:35.156332 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.155755    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:35.156332 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.155774    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:35.156332 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.155788    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v2ll7 for pod openshift-network-diagnostics/network-check-target-75k5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:35.156332 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.155844    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7 podName:01e6577b-6859-4947-9532-fc0b9edcf3cb nodeName:}" failed. No retries permitted until 2026-09-18 20:20:37.155826123 +0000 UTC m=+6.186752247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2ll7" (UniqueName: "kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7") pod "network-check-target-75k5k" (UID: "01e6577b-6859-4947-9532-fc0b9edcf3cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:35.374096 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.374049    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 20:15:33 +0000 UTC" deadline="2028-04-14 20:19:18.566430828 +0000 UTC"
Sep 18 20:20:35.374096 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.374093    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13775h58m43.192342339s"
Sep 18 20:20:35.456031 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.455117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:35.456031 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.455255    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:35.456031 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.455679    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:35.456031 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.455790    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:35.456031 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.455863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:35.456031 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:35.455933    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:35.479305 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.479222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z7w45" event={"ID":"d8e15a8a-8986-42a3-9cba-945c5b6894a4","Type":"ContainerStarted","Data":"8e20fa996d871c28f82a4f0cebc29597bcccb60640dddb44a625552538533f55"}
Sep 18 20:20:35.482905 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.482879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerStarted","Data":"ea7749f2e0cf4a62a6bd02e7ac92e94ce380cd70c8212ed4a30936e69382cc0b"}
Sep 18 20:20:35.496304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.496248    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vzx2c" event={"ID":"7def255f-b389-4260-b3bd-cb1ce422bef6","Type":"ContainerStarted","Data":"d6200ce187ee46e0df56df69418ea256e9d6e54f01fff33eb6beb6a24d14890c"}
Sep 18 20:20:35.506355 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.506309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mh7w7" event={"ID":"325de808-8e01-460b-9507-866c7181e24d","Type":"ContainerStarted","Data":"1efb9ddb343ab69cdc88db7f1ff2fd319ba083d42ae17829c912c9c887250aa5"}
Sep 18 20:20:35.514409 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.514246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c" event={"ID":"dc154dc3-77eb-4bf4-a48a-daad8c7385bf","Type":"ContainerStarted","Data":"ec6bbdc2bb5ec884473fc4b83767692b4906550466d3cd6b163324048b36130c"}
Sep 18 20:20:35.520840 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.520096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal" event={"ID":"36964665b9e807b88bd927b05ac588ed","Type":"ContainerStarted","Data":"f86766f13a6651eabf29323a843782f39ecee283dc0e35899849ce95e8689f25"}
Sep 18 20:20:35.538829 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.538650    2566 generic.go:356] "Generic (PLEG): container finished" podID="2ed2e113751e9f2a9cf8de7dcd23788f" containerID="f5c4d747ad6ec36f9cf64298adaf52b13ae962c8e38d0389715dd8f062ce34e8" exitCode=0
Sep 18 20:20:35.538829 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.538761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal" event={"ID":"2ed2e113751e9f2a9cf8de7dcd23788f","Type":"ContainerDied","Data":"f5c4d747ad6ec36f9cf64298adaf52b13ae962c8e38d0389715dd8f062ce34e8"}
Sep 18 20:20:35.547100 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.547075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b7b5n" event={"ID":"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a","Type":"ContainerStarted","Data":"5be58ceab42b0910c670cfaf32730459c6596b338380251ae6d174f7dc2b3e60"}
Sep 18 20:20:35.555994 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.555936    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-76.ec2.internal" podStartSLOduration=3.555918236 podStartE2EDuration="3.555918236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:35.535888656 +0000 UTC m=+4.566814795" watchObservedRunningTime="2026-09-18 20:20:35.555918236 +0000 UTC m=+4.586844378"
Sep 18 20:20:35.559380 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.559354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2l4jx" event={"ID":"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52","Type":"ContainerStarted","Data":"5ac859be51155e45dcdc5759c2dff6188db19fc210d365bd32f1d9e05544e586"}
Sep 18 20:20:35.562029 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.561955    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hglx4" event={"ID":"d5396b35-d934-4cbd-bf7a-60e0ecb853f7","Type":"ContainerStarted","Data":"ac985710eecfaf329fe17de133c97dbee54b7460b81cc70cc7be1dc7ed03c943"}
Sep 18 20:20:35.573207 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:35.573184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8m8l" event={"ID":"2c273cf8-0881-435f-be53-928e52dd5a58","Type":"ContainerStarted","Data":"0c198bac66a7946af9ff017d89b6d63f9cf9da492c884675c67afcff48131e7b"}
Sep 18 20:20:36.602697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:36.602662    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal" event={"ID":"2ed2e113751e9f2a9cf8de7dcd23788f","Type":"ContainerStarted","Data":"e6224bfe8806e5098211c24c21215013a00dd8021303c5d4963c7bf645711f7a"}
Sep 18 20:20:37.070980 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:37.070944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:37.071165 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:37.071001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:37.071165 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.071138    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:37.071309 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.071199    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret podName:1a406b73-2c97-49f0-82b9-964afe466106 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:41.071181511 +0000 UTC m=+10.102107629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret") pod "global-pull-secret-syncer-msj7n" (UID: "1a406b73-2c97-49f0-82b9-964afe466106") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:37.071696 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.071674    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:37.071785 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.071740    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs podName:4e160c7e-4ea8-48e5-b932-32be44d1c289 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:41.071720724 +0000 UTC m=+10.102646845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs") pod "network-metrics-daemon-zfhl9" (UID: "4e160c7e-4ea8-48e5-b932-32be44d1c289") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:37.171499 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:37.171465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:37.171678 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.171641    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:37.171678 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.171663    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:37.171678 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.171676    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v2ll7 for pod openshift-network-diagnostics/network-check-target-75k5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:37.171843 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.171733    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7 podName:01e6577b-6859-4947-9532-fc0b9edcf3cb nodeName:}" failed. No retries permitted until 2026-09-18 20:20:41.171715604 +0000 UTC m=+10.202641750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2ll7" (UniqueName: "kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7") pod "network-check-target-75k5k" (UID: "01e6577b-6859-4947-9532-fc0b9edcf3cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:37.455999 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:37.455485    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:37.455999 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.455626    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:37.455999 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:37.455485    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:37.455999 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.455768    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:37.455999 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:37.455838    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:37.455999 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:37.455928    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:39.455582 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:39.455540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:39.456059 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:39.455681    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:39.456132 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:39.456088    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:39.456132 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:39.456096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:39.456240 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:39.456218    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:39.456400 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:39.456304    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:41.106323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:41.106284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:41.106786 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:41.106340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:41.106786 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.106419    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:41.106786 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.106438    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:41.106786 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.106493    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret podName:1a406b73-2c97-49f0-82b9-964afe466106 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:49.106473782 +0000 UTC m=+18.137399922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret") pod "global-pull-secret-syncer-msj7n" (UID: "1a406b73-2c97-49f0-82b9-964afe466106") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:41.106786 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.106514    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs podName:4e160c7e-4ea8-48e5-b932-32be44d1c289 nodeName:}" failed. No retries permitted until 2026-09-18 20:20:49.106503134 +0000 UTC m=+18.137429274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs") pod "network-metrics-daemon-zfhl9" (UID: "4e160c7e-4ea8-48e5-b932-32be44d1c289") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:41.207730 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:41.207599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:41.207946 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.207747    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:41.207946 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.207776    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:41.207946 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.207790    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v2ll7 for pod openshift-network-diagnostics/network-check-target-75k5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:41.207946 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.207853    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7 podName:01e6577b-6859-4947-9532-fc0b9edcf3cb nodeName:}" failed. No retries permitted until 2026-09-18 20:20:49.20783415 +0000 UTC m=+18.238760282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2ll7" (UniqueName: "kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7") pod "network-check-target-75k5k" (UID: "01e6577b-6859-4947-9532-fc0b9edcf3cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:41.457186 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:41.457112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:41.457336 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.457234    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:41.457585 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:41.457565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:41.457693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:41.457599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:41.457693 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.457681    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:41.457794 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:41.457770    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:43.455831 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:43.455410    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:43.455831 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:43.455415    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:43.455831 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:43.455528    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:43.455831 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:43.455628    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:43.455831 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:43.455417    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:43.455831 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:43.455713    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:45.457576 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:45.457542    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:45.457576 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:45.457566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:45.458057 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:45.457675    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:45.458057 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:45.457686    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:45.458057 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:45.457802    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:45.458057 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:45.457895    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:47.455102 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:47.455069    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:47.455546 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:47.455110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:47.455546 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:47.455186    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:47.455546 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:47.455070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:47.455546 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:47.455293    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:47.455546 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:47.455345    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:49.168529 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:49.168485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:49.169002 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:49.168549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:49.169002 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.168655    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:49.169002 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.168668    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:49.169002 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.168726    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret podName:1a406b73-2c97-49f0-82b9-964afe466106 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:05.168708032 +0000 UTC m=+34.199634151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret") pod "global-pull-secret-syncer-msj7n" (UID: "1a406b73-2c97-49f0-82b9-964afe466106") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:20:49.169002 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.168743    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs podName:4e160c7e-4ea8-48e5-b932-32be44d1c289 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:05.168733718 +0000 UTC m=+34.199659836 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs") pod "network-metrics-daemon-zfhl9" (UID: "4e160c7e-4ea8-48e5-b932-32be44d1c289") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:20:49.269961 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:49.269923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:49.270145 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.270093    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:20:49.270145 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.270113    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:20:49.270145 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.270127    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v2ll7 for pod openshift-network-diagnostics/network-check-target-75k5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:49.270317 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.270193    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7 podName:01e6577b-6859-4947-9532-fc0b9edcf3cb nodeName:}" failed. No retries permitted until 2026-09-18 20:21:05.270173582 +0000 UTC m=+34.301099703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2ll7" (UniqueName: "kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7") pod "network-check-target-75k5k" (UID: "01e6577b-6859-4947-9532-fc0b9edcf3cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:20:49.455151 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:49.455070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:49.455299 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:49.455070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:49.455299 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.455199    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:49.455299 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:49.455207    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:49.455427 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.455292    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:49.455427 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:49.455378    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:51.455886 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:51.455849    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:51.456324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:51.455949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:51.456324 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:51.455984    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:51.456324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:51.456053    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:51.456324 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:51.456053    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:51.456324 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:51.456152    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:52.635511 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.635335    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c273cf8-0881-435f-be53-928e52dd5a58" containerID="5b72cde3519a10578961a630d8eee17226b9a833322cae808ec390d8425f6d4a" exitCode=0
Sep 18 20:20:52.636230 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.635420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8m8l" event={"ID":"2c273cf8-0881-435f-be53-928e52dd5a58","Type":"ContainerDied","Data":"5b72cde3519a10578961a630d8eee17226b9a833322cae808ec390d8425f6d4a"}
Sep 18 20:20:52.638085 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.637951    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z7w45" event={"ID":"d8e15a8a-8986-42a3-9cba-945c5b6894a4","Type":"ContainerStarted","Data":"bde4588c6a54dd12e2b79c1b2e4a34c5f397eb02236caa6e64c20f141c71fb5c"}
Sep 18 20:20:52.640328 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.640312    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:20:52.640657 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.640639    2566 generic.go:356] "Generic (PLEG): container finished" podID="ce0abf55-e019-4d4b-8bab-45921f3645c2" containerID="4497fee09eb0644054e44b20b6d946d3791271fc3a33f4a05016c569ee517347" exitCode=1
Sep 18 20:20:52.640742 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.640700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerStarted","Data":"a970b95965639367cf0d810d739da6c43c2ee51153b8476c27311025108dda6d"}
Sep 18 20:20:52.640789 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.640754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerDied","Data":"4497fee09eb0644054e44b20b6d946d3791271fc3a33f4a05016c569ee517347"}
Sep 18 20:20:52.640847 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.640796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerStarted","Data":"2011e08c55e2c5af458982da6d09cb35c51dd341d4621befc01582ff21863118"}
Sep 18 20:20:52.642307 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.642291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vzx2c" event={"ID":"7def255f-b389-4260-b3bd-cb1ce422bef6","Type":"ContainerStarted","Data":"b5d7da09f40f6b016c027708ac21354be43f596de4b4ca7f620fa077ad8caeda"}
Sep 18 20:20:52.643851 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.643833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mh7w7" event={"ID":"325de808-8e01-460b-9507-866c7181e24d","Type":"ContainerStarted","Data":"051afb29eee61f385079a069cd3ad4a2f312f90eb5d6fc5c5436015f59c21178"}
Sep 18 20:20:52.645288 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.645267    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c" event={"ID":"dc154dc3-77eb-4bf4-a48a-daad8c7385bf","Type":"ContainerStarted","Data":"e78114537b0906c85aff6ad4d7e2a95688e95ffd61ee9044edfc8d260caff0b1"}
Sep 18 20:20:52.646825 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.646806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2l4jx" event={"ID":"b2f71b87-9bf3-454d-a2e1-97a0ffa97f52","Type":"ContainerStarted","Data":"e47223e93f09be3d76ffa0a02aa73e587667a4dda7827e57d9249a284ae5f088"}
Sep 18 20:20:52.648306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.648289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hglx4" event={"ID":"d5396b35-d934-4cbd-bf7a-60e0ecb853f7","Type":"ContainerStarted","Data":"89fbf042e729fcce40e5381453be19a4a1794d694d364c271c5bc0841008853a"}
Sep 18 20:20:52.657244 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.657204    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-76.ec2.internal" podStartSLOduration=20.65719216 podStartE2EDuration="20.65719216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:36.619742948 +0000 UTC m=+5.650669089" watchObservedRunningTime="2026-09-18 20:20:52.65719216 +0000 UTC m=+21.688118300"
Sep 18 20:20:52.677301 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.677249    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vzx2c" podStartSLOduration=4.098843812 podStartE2EDuration="21.677228504s" totalImagesPullingTime="17.578384692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:52.675956483 +0000 UTC m=+21.706882624" watchObservedRunningTime="2026-09-18 20:20:52.677228504 +0000 UTC m=+21.708154647"
Sep 18 20:20:52.694640 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.694603    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mh7w7" podStartSLOduration=11.400446306 podStartE2EDuration="21.694593913s" totalImagesPullingTime="10.294147607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:52.694192265 +0000 UTC m=+21.725118406" watchObservedRunningTime="2026-09-18 20:20:52.694593913 +0000 UTC m=+21.725520052"
Sep 18 20:20:52.712773 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.712734    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hglx4" podStartSLOduration=4.136835037 podStartE2EDuration="21.712719406s" totalImagesPullingTime="17.575884369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:52.712577498 +0000 UTC m=+21.743503637" watchObservedRunningTime="2026-09-18 20:20:52.712719406 +0000 UTC m=+21.743645546"
Sep 18 20:20:52.728101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.728060    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z7w45" podStartSLOduration=4.15722544 podStartE2EDuration="21.728045521s" totalImagesPullingTime="17.570820081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:52.727574157 +0000 UTC m=+21.758500296" watchObservedRunningTime="2026-09-18 20:20:52.728045521 +0000 UTC m=+21.758971661"
Sep 18 20:20:52.747136 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:52.747102    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2l4jx" podStartSLOduration=3.929178358 podStartE2EDuration="21.747086951s" totalImagesPullingTime="17.817908593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:52.746840836 +0000 UTC m=+21.777766977" watchObservedRunningTime="2026-09-18 20:20:52.747086951 +0000 UTC m=+21.778013089"
Sep 18 20:20:53.454960 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.454930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:53.454960 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.454957    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:53.455201 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:53.455069    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:53.455201 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:53.455151    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:53.455295 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.455219    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:53.455334 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:53.455305    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:53.515589 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.515564    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:53.516354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.516326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:53.654408 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.654383    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:20:53.654906 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.654878    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerStarted","Data":"2b9103b2760532f105a7d530979e157c7edf7fa44a74cecfe54ed19626202077"}
Sep 18 20:20:53.654990 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.654968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerStarted","Data":"41cb8d32384d0cf4aaaa807304edec55bc67cad1fb360b7832b72118f0bed164"}
Sep 18 20:20:53.655118 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.655044    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerStarted","Data":"4a59f57f354380c108d9ed748f1120d77c72f54dae567753e51ce6c65b919396"}
Sep 18 20:20:53.655743 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.655654    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:53.656208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.656193    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mh7w7"
Sep 18 20:20:53.722509 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:53.722477    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 20:20:54.535906 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:54.535796    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T20:20:53.722500691Z","UUID":"bfdee86b-2502-436e-9645-57fa30ee7dc2","Handler":null,"Name":"","Endpoint":""}
Sep 18 20:20:54.537412 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:54.537389    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 20:20:54.537523 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:54.537424    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 20:20:54.659094 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:54.659049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b7b5n" event={"ID":"7cb8b8b3-f9f1-41eb-b7d0-d571159c441a","Type":"ContainerStarted","Data":"553b37ef768296326d749c31e8132e26b7b2a99b9591ee8ffd28395eed0737a6"}
Sep 18 20:20:54.661230 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:54.661192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c" event={"ID":"dc154dc3-77eb-4bf4-a48a-daad8c7385bf","Type":"ContainerStarted","Data":"3981a20a8b03f80b74e45a4b7157f9b06fcca6b79689ca5d0886c45b7173637c"}
Sep 18 20:20:55.455388 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:55.455355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:55.455388 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:55.455371    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:55.455632 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:55.455485    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:55.455632 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:55.455486    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:55.455632 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:55.455591    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:55.455785 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:55.455669    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:55.667618 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:55.667590    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:20:55.668069 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:55.668045    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerStarted","Data":"e565458a07bd3aa42fccd4b08518650346e08ca644a5effa1f378264974b394e"}
Sep 18 20:20:55.670492 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:55.670413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c" event={"ID":"dc154dc3-77eb-4bf4-a48a-daad8c7385bf","Type":"ContainerStarted","Data":"33ce8feb5500a61839214003da49aa9a7b9fe678b82c3ec923dfc57e413c4bea"}
Sep 18 20:20:55.687697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:55.687648    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b7b5n" podStartSLOduration=7.116835601 podStartE2EDuration="24.687632292s" totalImagesPullingTime="17.570796691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:54.67397121 +0000 UTC m=+23.704897350" watchObservedRunningTime="2026-09-18 20:20:55.687632292 +0000 UTC m=+24.718558431"
Sep 18 20:20:55.687822 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:55.687730    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dz75c" podStartSLOduration=4.921650785 podStartE2EDuration="24.687724244s" totalImagesPullingTime="19.766073459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:55.687396803 +0000 UTC m=+24.718322945" watchObservedRunningTime="2026-09-18 20:20:55.687724244 +0000 UTC m=+24.718650402"
Sep 18 20:20:57.454864 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.454690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:57.455531 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.454709    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:57.455531 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:57.454931    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:20:57.455531 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:57.455041    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:57.455531 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.454771    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:57.455531 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:57.455142    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:57.678406 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.678385    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:20:57.678786 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.678765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerStarted","Data":"11dee9b953c0ed74bf7f26c1abdaad7a07d55fe805b883331b1332a03bef54ec"}
Sep 18 20:20:57.678996 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.678977    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:57.679283 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.679264    2566 scope.go:117] "RemoveContainer" containerID="4497fee09eb0644054e44b20b6d946d3791271fc3a33f4a05016c569ee517347"
Sep 18 20:20:57.680810 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.680783    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c273cf8-0881-435f-be53-928e52dd5a58" containerID="2d3db4a1e348e667d77dc87d0f2d5e77b9990040014c368d5c05e90e7807c4a5" exitCode=0
Sep 18 20:20:57.680910 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.680823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8m8l" event={"ID":"2c273cf8-0881-435f-be53-928e52dd5a58","Type":"ContainerDied","Data":"2d3db4a1e348e667d77dc87d0f2d5e77b9990040014c368d5c05e90e7807c4a5"}
Sep 18 20:20:57.699946 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:57.699929    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:58.686157 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:58.685966    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c273cf8-0881-435f-be53-928e52dd5a58" containerID="9e7f6b4a64deeaf23d9e66fcb04d2f669ce6c10cb06d178f743a5302822f575c" exitCode=0
Sep 18 20:20:58.686618 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:58.686043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8m8l" event={"ID":"2c273cf8-0881-435f-be53-928e52dd5a58","Type":"ContainerDied","Data":"9e7f6b4a64deeaf23d9e66fcb04d2f669ce6c10cb06d178f743a5302822f575c"}
Sep 18 20:20:58.690320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:58.690302    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:20:58.690736 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:58.690710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" event={"ID":"ce0abf55-e019-4d4b-8bab-45921f3645c2","Type":"ContainerStarted","Data":"07bfe4a317df32e53d037e9f6c17ffb3ebed272dd04d67b19cae37d1dbb73d81"}
Sep 18 20:20:58.690970 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:58.690955    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:58.691066 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:58.691027    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:58.706687 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:58.706604    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:20:58.740967 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:58.740924    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp" podStartSLOduration=10.145998313 podStartE2EDuration="27.740909077s" totalImagesPullingTime="17.594910764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:58.740429187 +0000 UTC m=+27.771355326" watchObservedRunningTime="2026-09-18 20:20:58.740909077 +0000 UTC m=+27.771835217"
Sep 18 20:20:59.026364 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:59.026333    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfhl9"]
Sep 18 20:20:59.026565 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:59.026477    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:20:59.026625 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:59.026595    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:20:59.029329 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:59.029301    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-msj7n"]
Sep 18 20:20:59.029475 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:59.029408    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:20:59.029539 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:59.029498    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:20:59.030056 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:59.030031    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-75k5k"]
Sep 18 20:20:59.030217 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:20:59.030126    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:20:59.030281 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:20:59.030215    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:21:00.454960 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:00.454931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:21:00.455388 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:00.454930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:21:00.455388 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:00.455050    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:21:00.455388 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:00.455128    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:21:00.699072 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:00.699033    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c273cf8-0881-435f-be53-928e52dd5a58" containerID="a45e85fff20a12a138d2a8e2d1fbf72397e59865df93ba8463bf809e354fbd1d" exitCode=0
Sep 18 20:21:00.699218 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:00.699073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8m8l" event={"ID":"2c273cf8-0881-435f-be53-928e52dd5a58","Type":"ContainerDied","Data":"a45e85fff20a12a138d2a8e2d1fbf72397e59865df93ba8463bf809e354fbd1d"}
Sep 18 20:21:01.456001 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:01.455926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:21:01.456428 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:01.456058    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:21:02.455341 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:02.455149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:21:02.455528 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:02.455204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:21:02.455528 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:02.455421    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:21:02.455528 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:02.455513    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:21:03.454874 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:03.454841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:21:03.455307 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:03.454990    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:21:04.457946 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:04.457916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:21:04.458402 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:04.458376    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msj7n" podUID="1a406b73-2c97-49f0-82b9-964afe466106"
Sep 18 20:21:04.458485 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:04.458425    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:21:04.458552 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:04.458531    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75k5k" podUID="01e6577b-6859-4947-9532-fc0b9edcf3cb"
Sep 18 20:21:05.181647 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:05.181624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:21:05.181752 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:05.181661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:21:05.181789 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:05.181767    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:05.181822 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:05.181814    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret podName:1a406b73-2c97-49f0-82b9-964afe466106 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:37.18180201 +0000 UTC m=+66.212728129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret") pod "global-pull-secret-syncer-msj7n" (UID: "1a406b73-2c97-49f0-82b9-964afe466106") : object "kube-system"/"original-pull-secret" not registered
Sep 18 20:21:05.181864 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:05.181767    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:05.181894 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:05.181885    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs podName:4e160c7e-4ea8-48e5-b932-32be44d1c289 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:37.181873035 +0000 UTC m=+66.212799167 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs") pod "network-metrics-daemon-zfhl9" (UID: "4e160c7e-4ea8-48e5-b932-32be44d1c289") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 20:21:05.282880 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:05.282849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:21:05.283043 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:05.283027    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 20:21:05.283105 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:05.283047    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 20:21:05.283105 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:05.283058    2566 projected.go:196] Error preparing data for projected volume kube-api-access-v2ll7 for pod openshift-network-diagnostics/network-check-target-75k5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:05.283174 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:05.283108    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7 podName:01e6577b-6859-4947-9532-fc0b9edcf3cb nodeName:}" failed. No retries permitted until 2026-09-18 20:21:37.283095237 +0000 UTC m=+66.314021355 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2ll7" (UniqueName: "kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7") pod "network-check-target-75k5k" (UID: "01e6577b-6859-4947-9532-fc0b9edcf3cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 20:21:05.334600 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:05.334555    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hglx4_d5396b35-d934-4cbd-bf7a-60e0ecb853f7/dns-node-resolver/0.log"
Sep 18 20:21:05.455023 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:05.454985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:21:05.455165 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:05.455106    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfhl9" podUID="4e160c7e-4ea8-48e5-b932-32be44d1c289"
Sep 18 20:21:05.713101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:05.713001    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c273cf8-0881-435f-be53-928e52dd5a58" containerID="d9dc31b1668eaf0294f3a740ebda76fa01d5ad1e12ce20772159fef3c2135774" exitCode=0
Sep 18 20:21:05.713101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:05.713045    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8m8l" event={"ID":"2c273cf8-0881-435f-be53-928e52dd5a58","Type":"ContainerDied","Data":"d9dc31b1668eaf0294f3a740ebda76fa01d5ad1e12ce20772159fef3c2135774"}
Sep 18 20:21:06.241660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.241633    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-76.ec2.internal" event="NodeReady"
Sep 18 20:21:06.241822 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.241743    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 20:21:06.274501 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.274475    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cc8799c58-s2j4f"]
Sep 18 20:21:06.307379 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.307353    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dwvtg"]
Sep 18 20:21:06.307513 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.307498    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.310304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.310279    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 20:21:06.310536 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.310512    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ptsqr\""
Sep 18 20:21:06.310798 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.310652    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 20:21:06.310919 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.310901    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 20:21:06.321370 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.321351    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 20:21:06.327465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.327446    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xvxl9"]
Sep 18 20:21:06.327592 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.327577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwvtg"
Sep 18 20:21:06.330367 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.330344    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 20:21:06.330460 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.330375    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 20:21:06.330460 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.330394    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 20:21:06.330552 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.330478    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cgdxn\""
Sep 18 20:21:06.346522 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.346502    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cc8799c58-s2j4f"]
Sep 18 20:21:06.346600 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.346561    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwvtg"]
Sep 18 20:21:06.346600 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.346577    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvxl9"]
Sep 18 20:21:06.346600 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.346504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.346692 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.346609    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-phhxn"]
Sep 18 20:21:06.348859 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.348838    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 20:21:06.349267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.349247    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 20:21:06.349349 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.349317    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s6fww\""
Sep 18 20:21:06.349412 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.349345    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 20:21:06.349412 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.349354    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 20:21:06.367167 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.367146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-phhxn"]
Sep 18 20:21:06.367242 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.367229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.369954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.369939    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 20:21:06.370066 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.369964    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 20:21:06.370066 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.370000    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l94f7\""
Sep 18 20:21:06.391387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.391367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-certificates\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.391487 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.391403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqlz\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-kube-api-access-xtqlz\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.391487 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.391429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-image-registry-private-configuration\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.391487 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.391466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.391597 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.391491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1eb0f128-f9ca-49be-ac71-204992e694f4-ca-trust-extracted\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.391597 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.391512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-installation-pull-secrets\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.391597 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.391537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-bound-sa-token\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.391597 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.391558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-trusted-ca\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.455044 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.454999    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:21:06.455225 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.455207    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:21:06.457886 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.457868    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5lg4r\""
Sep 18 20:21:06.457999 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.457883    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 20:21:06.457999 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.457869    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 20:21:06.457999 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.457921    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 20:21:06.491989 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.491945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqlz\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-kube-api-access-xtqlz\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.491989 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.491970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-image-registry-private-configuration\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.491989 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.491987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.492140 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1eb0f128-f9ca-49be-ac71-204992e694f4-ca-trust-extracted\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.492140 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f0c548-516a-4413-b7f5-7012a0423ff1-data-volume\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.492214 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-installation-pull-secrets\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.492214 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.492173    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:21:06.492214 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.492190    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc8799c58-s2j4f: secret "image-registry-tls" not found
Sep 18 20:21:06.492214 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-bound-sa-token\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.492399 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cs44\" (UniqueName: \"kubernetes.io/projected/82f87074-724c-4989-a468-5a5287a130e4-kube-api-access-8cs44\") pod \"ingress-canary-dwvtg\" (UID: \"82f87074-724c-4989-a468-5a5287a130e4\") " pod="openshift-ingress-canary/ingress-canary-dwvtg"
Sep 18 20:21:06.492399 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-trusted-ca\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.492399 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.492399 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1eb0f128-f9ca-49be-ac71-204992e694f4-ca-trust-extracted\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.492399 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82f87074-724c-4989-a468-5a5287a130e4-cert\") pod \"ingress-canary-dwvtg\" (UID: \"82f87074-724c-4989-a468-5a5287a130e4\") " pod="openshift-ingress-canary/ingress-canary-dwvtg"
Sep 18 20:21:06.492399 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.492391    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls podName:1eb0f128-f9ca-49be-ac71-204992e694f4 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:06.992369547 +0000 UTC m=+36.023295680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls") pod "image-registry-7cc8799c58-s2j4f" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4") : secret "image-registry-tls" not found
Sep 18 20:21:06.492679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85f0c548-516a-4413-b7f5-7012a0423ff1-crio-socket\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.492679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbmg5\" (UniqueName: \"kubernetes.io/projected/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-api-access-xbmg5\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.492679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd081-7eaa-4ab3-beb0-48e17424b080-tmp-dir\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.492679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-certificates\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.492679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492583    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.492679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.492679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0cd081-7eaa-4ab3-beb0-48e17424b080-config-volume\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.492901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.492676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzf2\" (UniqueName: \"kubernetes.io/projected/0c0cd081-7eaa-4ab3-beb0-48e17424b080-kube-api-access-zmzf2\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.493092 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.493073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-certificates\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.493146 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.493108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-trusted-ca\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.495838 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.495817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-installation-pull-secrets\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.495909 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.495888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-image-registry-private-configuration\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.502596 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.502576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-bound-sa-token\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.502680 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.502640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqlz\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-kube-api-access-xtqlz\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.593998 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.593974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f0c548-516a-4413-b7f5-7012a0423ff1-data-volume\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.594144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cs44\" (UniqueName: \"kubernetes.io/projected/82f87074-724c-4989-a468-5a5287a130e4-kube-api-access-8cs44\") pod \"ingress-canary-dwvtg\" (UID: \"82f87074-724c-4989-a468-5a5287a130e4\") " pod="openshift-ingress-canary/ingress-canary-dwvtg"
Sep 18 20:21:06.594144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.594144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82f87074-724c-4989-a468-5a5287a130e4-cert\") pod \"ingress-canary-dwvtg\" (UID: \"82f87074-724c-4989-a468-5a5287a130e4\") " pod="openshift-ingress-canary/ingress-canary-dwvtg"
Sep 18 20:21:06.594144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85f0c548-516a-4413-b7f5-7012a0423ff1-crio-socket\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.594144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbmg5\" (UniqueName: \"kubernetes.io/projected/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-api-access-xbmg5\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd081-7eaa-4ab3-beb0-48e17424b080-tmp-dir\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0cd081-7eaa-4ab3-beb0-48e17424b080-config-volume\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzf2\" (UniqueName: \"kubernetes.io/projected/0c0cd081-7eaa-4ab3-beb0-48e17424b080-kube-api-access-zmzf2\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.594321    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85f0c548-516a-4413-b7f5-7012a0423ff1-crio-socket\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.594360    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f0c548-516a-4413-b7f5-7012a0423ff1-data-volume\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.594387 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.594380    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:07.094362472 +0000 UTC m=+36.125288612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:06.594855 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.594415    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:07.094398194 +0000 UTC m=+36.125324333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:06.594855 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.594436    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:21:06.594855 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.594481    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls podName:0c0cd081-7eaa-4ab3-beb0-48e17424b080 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:07.094462147 +0000 UTC m=+36.125388269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls") pod "dns-default-phhxn" (UID: "0c0cd081-7eaa-4ab3-beb0-48e17424b080") : secret "dns-default-metrics-tls" not found
Sep 18 20:21:06.594855 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c0cd081-7eaa-4ab3-beb0-48e17424b080-tmp-dir\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.594855 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.594820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0cd081-7eaa-4ab3-beb0-48e17424b080-config-volume\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.596554 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.596531    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82f87074-724c-4989-a468-5a5287a130e4-cert\") pod \"ingress-canary-dwvtg\" (UID: \"82f87074-724c-4989-a468-5a5287a130e4\") " pod="openshift-ingress-canary/ingress-canary-dwvtg"
Sep 18 20:21:06.604170 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.604144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzf2\" (UniqueName: \"kubernetes.io/projected/0c0cd081-7eaa-4ab3-beb0-48e17424b080-kube-api-access-zmzf2\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:06.604423 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.604404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbmg5\" (UniqueName: \"kubernetes.io/projected/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-api-access-xbmg5\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:06.604634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.604618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cs44\" (UniqueName: \"kubernetes.io/projected/82f87074-724c-4989-a468-5a5287a130e4-kube-api-access-8cs44\") pod \"ingress-canary-dwvtg\" (UID: \"82f87074-724c-4989-a468-5a5287a130e4\") " pod="openshift-ingress-canary/ingress-canary-dwvtg"
Sep 18 20:21:06.627998 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.627974    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-g2whk"]
Sep 18 20:21:06.630174 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.630160    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.632802 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.632781    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 20:21:06.632887 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.632801    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cskv4\""
Sep 18 20:21:06.632887 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.632807    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 20:21:06.632887 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.632807    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 20:21:06.632887 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.632877    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 20:21:06.635357 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.635342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwvtg"
Sep 18 20:21:06.639355 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.639336    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-g2whk"]
Sep 18 20:21:06.710301 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.710185    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z7w45_d8e15a8a-8986-42a3-9cba-945c5b6894a4/node-ca/0.log"
Sep 18 20:21:06.717797 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.717769    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c273cf8-0881-435f-be53-928e52dd5a58" containerID="54a89738ee25dcf36b6cad30d4a473b1fe8a378a0cc450bc492843cf57d87a4d" exitCode=0
Sep 18 20:21:06.718332 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.717823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8m8l" event={"ID":"2c273cf8-0881-435f-be53-928e52dd5a58","Type":"ContainerDied","Data":"54a89738ee25dcf36b6cad30d4a473b1fe8a378a0cc450bc492843cf57d87a4d"}
Sep 18 20:21:06.796766 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.796741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg2f5\" (UniqueName: \"kubernetes.io/projected/05eb5c25-c2db-4847-aa85-e2af6cd967eb-kube-api-access-qg2f5\") pod \"service-ca-55f6fc7b86-g2whk\" (UID: \"05eb5c25-c2db-4847-aa85-e2af6cd967eb\") " pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.796862 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.796801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/05eb5c25-c2db-4847-aa85-e2af6cd967eb-signing-cabundle\") pod \"service-ca-55f6fc7b86-g2whk\" (UID: \"05eb5c25-c2db-4847-aa85-e2af6cd967eb\") " pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.796903 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.796862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/05eb5c25-c2db-4847-aa85-e2af6cd967eb-signing-key\") pod \"service-ca-55f6fc7b86-g2whk\" (UID: \"05eb5c25-c2db-4847-aa85-e2af6cd967eb\") " pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.897956 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.897928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/05eb5c25-c2db-4847-aa85-e2af6cd967eb-signing-cabundle\") pod \"service-ca-55f6fc7b86-g2whk\" (UID: \"05eb5c25-c2db-4847-aa85-e2af6cd967eb\") " pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.898117 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.898047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/05eb5c25-c2db-4847-aa85-e2af6cd967eb-signing-key\") pod \"service-ca-55f6fc7b86-g2whk\" (UID: \"05eb5c25-c2db-4847-aa85-e2af6cd967eb\") " pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.898117 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.898092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2f5\" (UniqueName: \"kubernetes.io/projected/05eb5c25-c2db-4847-aa85-e2af6cd967eb-kube-api-access-qg2f5\") pod \"service-ca-55f6fc7b86-g2whk\" (UID: \"05eb5c25-c2db-4847-aa85-e2af6cd967eb\") " pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.898717 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.898695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/05eb5c25-c2db-4847-aa85-e2af6cd967eb-signing-cabundle\") pod \"service-ca-55f6fc7b86-g2whk\" (UID: \"05eb5c25-c2db-4847-aa85-e2af6cd967eb\") " pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.901005 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.900987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/05eb5c25-c2db-4847-aa85-e2af6cd967eb-signing-key\") pod \"service-ca-55f6fc7b86-g2whk\" (UID: \"05eb5c25-c2db-4847-aa85-e2af6cd967eb\") " pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.905953 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.905933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2f5\" (UniqueName: \"kubernetes.io/projected/05eb5c25-c2db-4847-aa85-e2af6cd967eb-kube-api-access-qg2f5\") pod \"service-ca-55f6fc7b86-g2whk\" (UID: \"05eb5c25-c2db-4847-aa85-e2af6cd967eb\") " pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.910323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.910301    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwvtg"]
Sep 18 20:21:06.913708 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:06.913684    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f87074_724c_4989_a468_5a5287a130e4.slice/crio-6aa6fdaec7a7ca09e8aec0a61e4725e160da345c1338954a033c929e65e96905 WatchSource:0}: Error finding container 6aa6fdaec7a7ca09e8aec0a61e4725e160da345c1338954a033c929e65e96905: Status 404 returned error can't find the container with id 6aa6fdaec7a7ca09e8aec0a61e4725e160da345c1338954a033c929e65e96905
Sep 18 20:21:06.938508 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.938488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk"
Sep 18 20:21:06.999504 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:06.999467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:06.999618 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.999591    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:21:06.999618 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.999608    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc8799c58-s2j4f: secret "image-registry-tls" not found
Sep 18 20:21:06.999683 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:06.999659    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls podName:1eb0f128-f9ca-49be-ac71-204992e694f4 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:07.999645151 +0000 UTC m=+37.030571274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls") pod "image-registry-7cc8799c58-s2j4f" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4") : secret "image-registry-tls" not found
Sep 18 20:21:07.100354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.100274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:07.100354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.100306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:07.100526 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.100360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:07.100526 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:07.100418    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:07.100526 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:07.100459    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:07.100526 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:07.100474    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:21:07.100526 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:07.100489    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:08.100472203 +0000 UTC m=+37.131398343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:07.100526 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:07.100513    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls podName:0c0cd081-7eaa-4ab3-beb0-48e17424b080 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:08.100498276 +0000 UTC m=+37.131424393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls") pod "dns-default-phhxn" (UID: "0c0cd081-7eaa-4ab3-beb0-48e17424b080") : secret "dns-default-metrics-tls" not found
Sep 18 20:21:07.100526 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:07.100526    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:08.100520522 +0000 UTC m=+37.131446640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:07.211525 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.211493    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-g2whk"]
Sep 18 20:21:07.215229 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:07.215208    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05eb5c25_c2db_4847_aa85_e2af6cd967eb.slice/crio-8ec1861f39a7d299908370b2cdfb982f7fe81cfa432baf5de76a106474fb2184 WatchSource:0}: Error finding container 8ec1861f39a7d299908370b2cdfb982f7fe81cfa432baf5de76a106474fb2184: Status 404 returned error can't find the container with id 8ec1861f39a7d299908370b2cdfb982f7fe81cfa432baf5de76a106474fb2184
Sep 18 20:21:07.455187 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.455090    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:21:07.457976 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.457947    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 20:21:07.458228 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.458214    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ls8xl\""
Sep 18 20:21:07.722468 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.722382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk" event={"ID":"05eb5c25-c2db-4847-aa85-e2af6cd967eb","Type":"ContainerStarted","Data":"8ec1861f39a7d299908370b2cdfb982f7fe81cfa432baf5de76a106474fb2184"}
Sep 18 20:21:07.727321 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.727144    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8m8l" event={"ID":"2c273cf8-0881-435f-be53-928e52dd5a58","Type":"ContainerStarted","Data":"551e25b5f4459bf479efe9929e18d360fe29ffc0b53fc0b91a8f128ffbcb5a3d"}
Sep 18 20:21:07.728716 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.728686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwvtg" event={"ID":"82f87074-724c-4989-a468-5a5287a130e4","Type":"ContainerStarted","Data":"6aa6fdaec7a7ca09e8aec0a61e4725e160da345c1338954a033c929e65e96905"}
Sep 18 20:21:07.763506 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:07.763432    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k8m8l" podStartSLOduration=8.702247519 podStartE2EDuration="36.763413162s" totalImagesPullingTime="28.061165643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:07.763208852 +0000 UTC m=+36.794134994" watchObservedRunningTime="2026-09-18 20:21:07.763413162 +0000 UTC m=+36.794339305"
Sep 18 20:21:08.008556 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:08.008472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:08.008702 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:08.008633    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:21:08.008702 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:08.008655    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc8799c58-s2j4f: secret "image-registry-tls" not found
Sep 18 20:21:08.008820 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:08.008719    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls podName:1eb0f128-f9ca-49be-ac71-204992e694f4 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:10.008695528 +0000 UTC m=+39.039621666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls") pod "image-registry-7cc8799c58-s2j4f" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4") : secret "image-registry-tls" not found
Sep 18 20:21:08.109429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:08.109388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:08.109598 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:08.109502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:08.109598 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:08.109538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:08.109598 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:08.109571    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:21:08.109744 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:08.109646    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls podName:0c0cd081-7eaa-4ab3-beb0-48e17424b080 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:10.109623638 +0000 UTC m=+39.140549771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls") pod "dns-default-phhxn" (UID: "0c0cd081-7eaa-4ab3-beb0-48e17424b080") : secret "dns-default-metrics-tls" not found
Sep 18 20:21:08.109744 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:08.109670    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:08.109744 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:08.109729    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:10.109713612 +0000 UTC m=+39.140639752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:08.109901 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:08.109802    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:08.109901 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:08.109839    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:10.109827298 +0000 UTC m=+39.140753419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:09.735170 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:09.734914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwvtg" event={"ID":"82f87074-724c-4989-a468-5a5287a130e4","Type":"ContainerStarted","Data":"620ca5aee2a8d04eb0a1d3f1320070b2be6a1c15e1eb95b7d98a660a8c8245ba"}
Sep 18 20:21:09.741222 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:09.741187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk" event={"ID":"05eb5c25-c2db-4847-aa85-e2af6cd967eb","Type":"ContainerStarted","Data":"2304a0b3df5eb4c2176980c32e9ca187b5da114ac34d3ca93dd0e379c2b3a5b1"}
Sep 18 20:21:09.753491 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:09.753441    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dwvtg" podStartSLOduration=1.151080901 podStartE2EDuration="3.753427299s" totalImagesPullingTime="2.602346398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:09.75308421 +0000 UTC m=+38.784010345" watchObservedRunningTime="2026-09-18 20:21:09.753427299 +0000 UTC m=+38.784353442"
Sep 18 20:21:09.769482 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:09.769364    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-g2whk" podStartSLOduration=1.46845221 podStartE2EDuration="3.769351778s" totalImagesPullingTime="2.300899568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:09.768931096 +0000 UTC m=+38.799857237" watchObservedRunningTime="2026-09-18 20:21:09.769351778 +0000 UTC m=+38.800277923"
Sep 18 20:21:10.024052 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:10.023942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:10.024225 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:10.024110    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:21:10.024225 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:10.024132    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc8799c58-s2j4f: secret "image-registry-tls" not found
Sep 18 20:21:10.024225 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:10.024199    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls podName:1eb0f128-f9ca-49be-ac71-204992e694f4 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:14.024179859 +0000 UTC m=+43.055105994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls") pod "image-registry-7cc8799c58-s2j4f" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4") : secret "image-registry-tls" not found
Sep 18 20:21:10.124508 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:10.124472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:10.124675 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:10.124611    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:21:10.124675 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:10.124642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:10.124675 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:10.124666    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls podName:0c0cd081-7eaa-4ab3-beb0-48e17424b080 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:14.12465194 +0000 UTC m=+43.155578074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls") pod "dns-default-phhxn" (UID: "0c0cd081-7eaa-4ab3-beb0-48e17424b080") : secret "dns-default-metrics-tls" not found
Sep 18 20:21:10.124842 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:10.124697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:10.124842 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:10.124727    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:10.124842 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:10.124778    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:14.124762394 +0000 UTC m=+43.155688531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:10.124842 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:10.124795    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:10.124842 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:10.124832    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:14.124822386 +0000 UTC m=+43.155748527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:14.056471 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:14.056430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:14.056846 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:14.056608    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 20:21:14.056846 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:14.056628    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cc8799c58-s2j4f: secret "image-registry-tls" not found
Sep 18 20:21:14.056846 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:14.056704    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls podName:1eb0f128-f9ca-49be-ac71-204992e694f4 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:22.056686672 +0000 UTC m=+51.087612790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls") pod "image-registry-7cc8799c58-s2j4f" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4") : secret "image-registry-tls" not found
Sep 18 20:21:14.157430 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:14.157392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:14.157612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:14.157476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:14.157612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:14.157506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:14.157612 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:14.157543    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 20:21:14.157612 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:14.157589    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 20:21:14.157612 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:14.157602    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:14.157812 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:14.157610    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls podName:0c0cd081-7eaa-4ab3-beb0-48e17424b080 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:22.157593199 +0000 UTC m=+51.188519330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls") pod "dns-default-phhxn" (UID: "0c0cd081-7eaa-4ab3-beb0-48e17424b080") : secret "dns-default-metrics-tls" not found
Sep 18 20:21:14.157812 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:14.157661    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:22.157648064 +0000 UTC m=+51.188574182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : configmap "kube-rbac-proxy" not found
Sep 18 20:21:14.157812 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:14.157671    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls podName:85f0c548-516a-4413-b7f5-7012a0423ff1 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:22.157665942 +0000 UTC m=+51.188592060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvxl9" (UID: "85f0c548-516a-4413-b7f5-7012a0423ff1") : secret "insights-runtime-extractor-tls" not found
Sep 18 20:21:22.116893 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.116859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:22.120123 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.120100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") pod \"image-registry-7cc8799c58-s2j4f\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") " pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:22.217370 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.217338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:22.217370 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.217372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:22.217578 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.217411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:22.217788 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.217767    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:22.217957 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.217934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f0c548-516a-4413-b7f5-7012a0423ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:22.219605 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.219585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f0c548-516a-4413-b7f5-7012a0423ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvxl9\" (UID: \"85f0c548-516a-4413-b7f5-7012a0423ff1\") " pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:22.219697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.219681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c0cd081-7eaa-4ab3-beb0-48e17424b080-metrics-tls\") pod \"dns-default-phhxn\" (UID: \"0c0cd081-7eaa-4ab3-beb0-48e17424b080\") " pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:22.255100 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.255073    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvxl9"
Sep 18 20:21:22.274344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.274278    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:22.505536 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.505513    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cc8799c58-s2j4f"]
Sep 18 20:21:22.507599 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:22.507571    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb0f128_f9ca_49be_ac71_204992e694f4.slice/crio-6fe353813e7eb8a6e0ca11708380bf841399f0b2e39bc70c868ab73c952e9eac WatchSource:0}: Error finding container 6fe353813e7eb8a6e0ca11708380bf841399f0b2e39bc70c868ab73c952e9eac: Status 404 returned error can't find the container with id 6fe353813e7eb8a6e0ca11708380bf841399f0b2e39bc70c868ab73c952e9eac
Sep 18 20:21:22.546215 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.546146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvxl9"]
Sep 18 20:21:22.556082 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:22.556060    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f0c548_516a_4413_b7f5_7012a0423ff1.slice/crio-368c26b2f80e5586ad12399ad4513d38798c21142521028cecd827e89e9cba66 WatchSource:0}: Error finding container 368c26b2f80e5586ad12399ad4513d38798c21142521028cecd827e89e9cba66: Status 404 returned error can't find the container with id 368c26b2f80e5586ad12399ad4513d38798c21142521028cecd827e89e9cba66
Sep 18 20:21:22.571415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.571395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-phhxn"]
Sep 18 20:21:22.573097 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:22.573072    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0cd081_7eaa_4ab3_beb0_48e17424b080.slice/crio-fd7bdaa4bec3eafd433ad9a4fd67d54997994819a9b74ee72dfaff91fbf387de WatchSource:0}: Error finding container fd7bdaa4bec3eafd433ad9a4fd67d54997994819a9b74ee72dfaff91fbf387de: Status 404 returned error can't find the container with id fd7bdaa4bec3eafd433ad9a4fd67d54997994819a9b74ee72dfaff91fbf387de
Sep 18 20:21:22.774631 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.774600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-phhxn" event={"ID":"0c0cd081-7eaa-4ab3-beb0-48e17424b080","Type":"ContainerStarted","Data":"fd7bdaa4bec3eafd433ad9a4fd67d54997994819a9b74ee72dfaff91fbf387de"}
Sep 18 20:21:22.776207 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.776187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvxl9" event={"ID":"85f0c548-516a-4413-b7f5-7012a0423ff1","Type":"ContainerStarted","Data":"a83b220472a44beca6706e9b48bdec3ffac5b4224ca3d35116d5f8c1bde96d44"}
Sep 18 20:21:22.776317 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.776229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvxl9" event={"ID":"85f0c548-516a-4413-b7f5-7012a0423ff1","Type":"ContainerStarted","Data":"368c26b2f80e5586ad12399ad4513d38798c21142521028cecd827e89e9cba66"}
Sep 18 20:21:22.777894 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.777873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f" event={"ID":"1eb0f128-f9ca-49be-ac71-204992e694f4","Type":"ContainerStarted","Data":"ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62"}
Sep 18 20:21:22.777983 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.777911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f" event={"ID":"1eb0f128-f9ca-49be-ac71-204992e694f4","Type":"ContainerStarted","Data":"6fe353813e7eb8a6e0ca11708380bf841399f0b2e39bc70c868ab73c952e9eac"}
Sep 18 20:21:22.778074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.778030    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:22.797946 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:22.797896    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f" podStartSLOduration=64.797882131 podStartE2EDuration="1m4.797882131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:22.796786121 +0000 UTC m=+51.827712262" watchObservedRunningTime="2026-09-18 20:21:22.797882131 +0000 UTC m=+51.828808273"
Sep 18 20:21:24.787323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:24.787288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-phhxn" event={"ID":"0c0cd081-7eaa-4ab3-beb0-48e17424b080","Type":"ContainerStarted","Data":"2ebecacc6013c20c9f73123cf04a225e959c81d5f775b1fd6c80ff250987cde2"}
Sep 18 20:21:24.787778 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:24.787354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-phhxn" event={"ID":"0c0cd081-7eaa-4ab3-beb0-48e17424b080","Type":"ContainerStarted","Data":"10e1911194e99b9fc97be47193c223ff6e186dd616922788544eaad094871834"}
Sep 18 20:21:24.787778 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:24.787398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:24.789534 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:24.789507    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvxl9" event={"ID":"85f0c548-516a-4413-b7f5-7012a0423ff1","Type":"ContainerStarted","Data":"107ce7beb2d89dd23f99a5f396bd83aa98da4794dbd4797b5ebaf7cd489542d1"}
Sep 18 20:21:24.806041 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:24.805980    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-phhxn" podStartSLOduration=17.202415637 podStartE2EDuration="18.805962219s" totalImagesPullingTime="1.603546582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:24.805576643 +0000 UTC m=+53.836502783" watchObservedRunningTime="2026-09-18 20:21:24.805962219 +0000 UTC m=+53.836888362"
Sep 18 20:21:25.796263 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:25.796228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvxl9" event={"ID":"85f0c548-516a-4413-b7f5-7012a0423ff1","Type":"ContainerStarted","Data":"ea99b0b573be34ab34fb23ba01e5be5b572db6903b42244e25df2ae0ed27ac16"}
Sep 18 20:21:25.816424 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:25.816384    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xvxl9" podStartSLOduration=16.884560025 podStartE2EDuration="19.816370393s" totalImagesPullingTime="2.931810368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:21:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:25.815931019 +0000 UTC m=+54.846857159" watchObservedRunningTime="2026-09-18 20:21:25.816370393 +0000 UTC m=+54.847296524"
Sep 18 20:21:28.034666 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.034638    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-5trl7"]
Sep 18 20:21:28.036421 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.036406    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-5trl7"
Sep 18 20:21:28.040084 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.040046    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-z584h\""
Sep 18 20:21:28.040084 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.040057    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 20:21:28.040914 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.040086    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 20:21:28.041550 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.041528    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"]
Sep 18 20:21:28.043314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.043294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"
Sep 18 20:21:28.045798 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.045779    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6hb4l\""
Sep 18 20:21:28.045959 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.045942    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 20:21:28.051222 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.051200    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cc8799c58-s2j4f"]
Sep 18 20:21:28.056468 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.056445    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-5trl7"]
Sep 18 20:21:28.059593 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.059571    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"]
Sep 18 20:21:28.062812 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.062758    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjq5h\" (UniqueName: \"kubernetes.io/projected/2f56239d-1c43-4f8d-905d-7f297ae92eeb-kube-api-access-hjq5h\") pod \"downloads-6594f4c75-5trl7\" (UID: \"2f56239d-1c43-4f8d-905d-7f297ae92eeb\") " pod="openshift-console/downloads-6594f4c75-5trl7"
Sep 18 20:21:28.062924 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.062862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fd2a4379-3d92-4f74-aabe-e5b380ea915d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-hk6kj\" (UID: \"fd2a4379-3d92-4f74-aabe-e5b380ea915d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"
Sep 18 20:21:28.136914 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.136883    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"]
Sep 18 20:21:28.138590 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.138575    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.157384 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.157362    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"]
Sep 18 20:21:28.163653 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.163633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5684f70-5e4f-41de-a375-5220cbb662b3-ca-trust-extracted\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.163753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.163662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5684f70-5e4f-41de-a375-5220cbb662b3-registry-certificates\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.163753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.163696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5684f70-5e4f-41de-a375-5220cbb662b3-image-registry-private-configuration\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.163833 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.163783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5684f70-5e4f-41de-a375-5220cbb662b3-bound-sa-token\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.163833 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.163808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn89f\" (UniqueName: \"kubernetes.io/projected/d5684f70-5e4f-41de-a375-5220cbb662b3-kube-api-access-gn89f\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.163833 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.163827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5684f70-5e4f-41de-a375-5220cbb662b3-trusted-ca\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.163984 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.163847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5684f70-5e4f-41de-a375-5220cbb662b3-registry-tls\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.163984 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.163928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5684f70-5e4f-41de-a375-5220cbb662b3-installation-pull-secrets\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.163984 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.163969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fd2a4379-3d92-4f74-aabe-e5b380ea915d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-hk6kj\" (UID: \"fd2a4379-3d92-4f74-aabe-e5b380ea915d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"
Sep 18 20:21:28.164147 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.164002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjq5h\" (UniqueName: \"kubernetes.io/projected/2f56239d-1c43-4f8d-905d-7f297ae92eeb-kube-api-access-hjq5h\") pod \"downloads-6594f4c75-5trl7\" (UID: \"2f56239d-1c43-4f8d-905d-7f297ae92eeb\") " pod="openshift-console/downloads-6594f4c75-5trl7"
Sep 18 20:21:28.166204 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.166180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fd2a4379-3d92-4f74-aabe-e5b380ea915d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-hk6kj\" (UID: \"fd2a4379-3d92-4f74-aabe-e5b380ea915d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"
Sep 18 20:21:28.177315 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.177288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjq5h\" (UniqueName: \"kubernetes.io/projected/2f56239d-1c43-4f8d-905d-7f297ae92eeb-kube-api-access-hjq5h\") pod \"downloads-6594f4c75-5trl7\" (UID: \"2f56239d-1c43-4f8d-905d-7f297ae92eeb\") " pod="openshift-console/downloads-6594f4c75-5trl7"
Sep 18 20:21:28.264875 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.264841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5684f70-5e4f-41de-a375-5220cbb662b3-ca-trust-extracted\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.265003 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.264884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5684f70-5e4f-41de-a375-5220cbb662b3-registry-certificates\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.265003 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.264918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5684f70-5e4f-41de-a375-5220cbb662b3-image-registry-private-configuration\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.265003 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.264973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5684f70-5e4f-41de-a375-5220cbb662b3-bound-sa-token\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.265003 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.264998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn89f\" (UniqueName: \"kubernetes.io/projected/d5684f70-5e4f-41de-a375-5220cbb662b3-kube-api-access-gn89f\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.265244 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.265070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5684f70-5e4f-41de-a375-5220cbb662b3-trusted-ca\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.265244 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.265119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5684f70-5e4f-41de-a375-5220cbb662b3-registry-tls\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.265244 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.265158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5684f70-5e4f-41de-a375-5220cbb662b3-installation-pull-secrets\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.265526 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.265507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5684f70-5e4f-41de-a375-5220cbb662b3-ca-trust-extracted\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.265885 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.265831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5684f70-5e4f-41de-a375-5220cbb662b3-registry-certificates\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.266237 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.266214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5684f70-5e4f-41de-a375-5220cbb662b3-trusted-ca\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.267786 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.267762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5684f70-5e4f-41de-a375-5220cbb662b3-registry-tls\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.267786 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.267781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5684f70-5e4f-41de-a375-5220cbb662b3-image-registry-private-configuration\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.267899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.267841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5684f70-5e4f-41de-a375-5220cbb662b3-installation-pull-secrets\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.274169 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.274144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5684f70-5e4f-41de-a375-5220cbb662b3-bound-sa-token\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.274480 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.274456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn89f\" (UniqueName: \"kubernetes.io/projected/d5684f70-5e4f-41de-a375-5220cbb662b3-kube-api-access-gn89f\") pod \"image-registry-5b6c9fd4b9-6stdj\" (UID: \"d5684f70-5e4f-41de-a375-5220cbb662b3\") " pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.348583 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.348520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-5trl7"
Sep 18 20:21:28.356383 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.354428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"
Sep 18 20:21:28.447390 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.447367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.622449 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.622356    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-5trl7"]
Sep 18 20:21:28.625939 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:28.625911    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f56239d_1c43_4f8d_905d_7f297ae92eeb.slice/crio-f32a0e48589c483db4cb2cff273820bd3d5a75167a3dab3d14a37c8734334526 WatchSource:0}: Error finding container f32a0e48589c483db4cb2cff273820bd3d5a75167a3dab3d14a37c8734334526: Status 404 returned error can't find the container with id f32a0e48589c483db4cb2cff273820bd3d5a75167a3dab3d14a37c8734334526
Sep 18 20:21:28.627376 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.627349    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"]
Sep 18 20:21:28.629632 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:28.629611    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2a4379_3d92_4f74_aabe_e5b380ea915d.slice/crio-0be885218eb1926e36eb3e9f4bb29030dcfe4f45f9f1d3c5f6565d15efac17e7 WatchSource:0}: Error finding container 0be885218eb1926e36eb3e9f4bb29030dcfe4f45f9f1d3c5f6565d15efac17e7: Status 404 returned error can't find the container with id 0be885218eb1926e36eb3e9f4bb29030dcfe4f45f9f1d3c5f6565d15efac17e7
Sep 18 20:21:28.718394 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.718375    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"]
Sep 18 20:21:28.720309 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:28.720282    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5684f70_5e4f_41de_a375_5220cbb662b3.slice/crio-7d1930893239acf44b8102a692e7ffdf06ac63e864ad6ad075a5be882246c1a9 WatchSource:0}: Error finding container 7d1930893239acf44b8102a692e7ffdf06ac63e864ad6ad075a5be882246c1a9: Status 404 returned error can't find the container with id 7d1930893239acf44b8102a692e7ffdf06ac63e864ad6ad075a5be882246c1a9
Sep 18 20:21:28.808246 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.808217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj" event={"ID":"d5684f70-5e4f-41de-a375-5220cbb662b3","Type":"ContainerStarted","Data":"33ff0e3901b3e6075ef2bd58f48d7844cd29f26b9412d9d8731638439bef66a0"}
Sep 18 20:21:28.808389 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.808303    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:28.808389 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.808327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj" event={"ID":"d5684f70-5e4f-41de-a375-5220cbb662b3","Type":"ContainerStarted","Data":"7d1930893239acf44b8102a692e7ffdf06ac63e864ad6ad075a5be882246c1a9"}
Sep 18 20:21:28.809551 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.809521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-5trl7" event={"ID":"2f56239d-1c43-4f8d-905d-7f297ae92eeb","Type":"ContainerStarted","Data":"f32a0e48589c483db4cb2cff273820bd3d5a75167a3dab3d14a37c8734334526"}
Sep 18 20:21:28.810671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.810640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj" event={"ID":"fd2a4379-3d92-4f74-aabe-e5b380ea915d","Type":"ContainerStarted","Data":"0be885218eb1926e36eb3e9f4bb29030dcfe4f45f9f1d3c5f6565d15efac17e7"}
Sep 18 20:21:28.827892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:28.827852    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj" podStartSLOduration=0.827839479 podStartE2EDuration="827.839479ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:28.827163992 +0000 UTC m=+57.858090129" watchObservedRunningTime="2026-09-18 20:21:28.827839479 +0000 UTC m=+57.858765613"
Sep 18 20:21:30.720464 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:30.720432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7xnhp"
Sep 18 20:21:30.819294 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:30.819259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj" event={"ID":"fd2a4379-3d92-4f74-aabe-e5b380ea915d","Type":"ContainerStarted","Data":"8ee2a6d803c494f6c7377665a84f6c2f7301248bdab98f7af6358f5e69aced30"}
Sep 18 20:21:30.819460 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:30.819398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"
Sep 18 20:21:30.824186 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:30.824162    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj"
Sep 18 20:21:30.835912 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:30.835876    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-hk6kj" podStartSLOduration=1.366873751 podStartE2EDuration="2.835860871s" totalImagesPullingTime="1.46898712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:30.835593364 +0000 UTC m=+59.866519507" watchObservedRunningTime="2026-09-18 20:21:30.835860871 +0000 UTC m=+59.866787012"
Sep 18 20:21:31.291913 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.291878    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"]
Sep 18 20:21:31.294459 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.294439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.297606 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.297582    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 20:21:31.297713 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.297582    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 20:21:31.298628 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.298608    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 20:21:31.298744 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.298719    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 20:21:31.299000 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.298983    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 20:21:31.299223 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.299204    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9mzng\""
Sep 18 20:21:31.310654 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.310627    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"]
Sep 18 20:21:31.392210 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.392175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9kw\" (UniqueName: \"kubernetes.io/projected/fddf80d0-1e99-4799-94e0-1415cbb122c4-kube-api-access-5b9kw\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.392372 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.392222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fddf80d0-1e99-4799-94e0-1415cbb122c4-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.392372 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.392290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddf80d0-1e99-4799-94e0-1415cbb122c4-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.392372 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.392329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fddf80d0-1e99-4799-94e0-1415cbb122c4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.493634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.493594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddf80d0-1e99-4799-94e0-1415cbb122c4-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.493789 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.493651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fddf80d0-1e99-4799-94e0-1415cbb122c4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.493789 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.493731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9kw\" (UniqueName: \"kubernetes.io/projected/fddf80d0-1e99-4799-94e0-1415cbb122c4-kube-api-access-5b9kw\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.493789 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.493763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fddf80d0-1e99-4799-94e0-1415cbb122c4-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.496539 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.496514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 20:21:31.496663 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.496560    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 20:21:31.496663 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.496566    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 20:21:31.502178 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.502158    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 20:21:31.504784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.504754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fddf80d0-1e99-4799-94e0-1415cbb122c4-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.506494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.506474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fddf80d0-1e99-4799-94e0-1415cbb122c4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.506672 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.506637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fddf80d0-1e99-4799-94e0-1415cbb122c4-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.512534 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.512509    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 20:21:31.522796 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.522773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9kw\" (UniqueName: \"kubernetes.io/projected/fddf80d0-1e99-4799-94e0-1415cbb122c4-kube-api-access-5b9kw\") pod \"prometheus-operator-7bd5d56d6-nmzkm\" (UID: \"fddf80d0-1e99-4799-94e0-1415cbb122c4\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.608350 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.608281    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9mzng\""
Sep 18 20:21:31.615620 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.615595    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"
Sep 18 20:21:31.894433 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:31.894363    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm"]
Sep 18 20:21:31.897983 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:31.897957    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddf80d0_1e99_4799_94e0_1415cbb122c4.slice/crio-2b5a05638ee720d95d58437a89c37a0ad12120c9c5ef3028891d88262bdf84f8 WatchSource:0}: Error finding container 2b5a05638ee720d95d58437a89c37a0ad12120c9c5ef3028891d88262bdf84f8: Status 404 returned error can't find the container with id 2b5a05638ee720d95d58437a89c37a0ad12120c9c5ef3028891d88262bdf84f8
Sep 18 20:21:32.403681 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.403647    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d4d6c48cb-wswgs"]
Sep 18 20:21:32.406321 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.405877    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.410374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.409915    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 20:21:32.410374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.410036    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 20:21:32.410374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.410056    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 20:21:32.410374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.409915    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 20:21:32.410374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.409918    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 20:21:32.410715 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.410391    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bqckb\""
Sep 18 20:21:32.417929 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.417903    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d4d6c48cb-wswgs"]
Sep 18 20:21:32.502118 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.502077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-serving-cert\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.502118 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.502120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-oauth-serving-cert\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.502343 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.502168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fzz\" (UniqueName: \"kubernetes.io/projected/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-kube-api-access-m9fzz\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.502343 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.502228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-oauth-config\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.502343 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.502321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-config\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.502493 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.502347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-service-ca\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.603664 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.603627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-serving-cert\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.603664 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.603674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-oauth-serving-cert\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.603898 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.603812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fzz\" (UniqueName: \"kubernetes.io/projected/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-kube-api-access-m9fzz\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.603898 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.603884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-oauth-config\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.604032 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.603940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-config\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.604032 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.603965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-service-ca\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.604480 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.604456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-oauth-serving-cert\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.604653 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.604628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-service-ca\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.604825 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.604780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-config\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.607063 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.607041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-oauth-config\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.608066 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.608043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-serving-cert\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.613007 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.612979    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fzz\" (UniqueName: \"kubernetes.io/projected/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-kube-api-access-m9fzz\") pod \"console-6d4d6c48cb-wswgs\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") " pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.719465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.719364    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:32.829446 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:32.829403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm" event={"ID":"fddf80d0-1e99-4799-94e0-1415cbb122c4","Type":"ContainerStarted","Data":"2b5a05638ee720d95d58437a89c37a0ad12120c9c5ef3028891d88262bdf84f8"}
Sep 18 20:21:33.008127 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:33.008039    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d4d6c48cb-wswgs"]
Sep 18 20:21:33.136774 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:33.136736    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfd42f5_7a25_4220_afd6_5162b6b3fb7e.slice/crio-9c02bd25994554ea9639501c0818eb494fc61e51ec9b3c5efc0048e422db0076 WatchSource:0}: Error finding container 9c02bd25994554ea9639501c0818eb494fc61e51ec9b3c5efc0048e422db0076: Status 404 returned error can't find the container with id 9c02bd25994554ea9639501c0818eb494fc61e51ec9b3c5efc0048e422db0076
Sep 18 20:21:33.835945 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:33.835750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4d6c48cb-wswgs" event={"ID":"acfd42f5-7a25-4220-afd6-5162b6b3fb7e","Type":"ContainerStarted","Data":"9c02bd25994554ea9639501c0818eb494fc61e51ec9b3c5efc0048e422db0076"}
Sep 18 20:21:33.838561 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:33.838527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm" event={"ID":"fddf80d0-1e99-4799-94e0-1415cbb122c4","Type":"ContainerStarted","Data":"a40578f620337d87950896c6f276d670aa50eba1af2b570f5045a0bf8bd35640"}
Sep 18 20:21:33.838685 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:33.838586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm" event={"ID":"fddf80d0-1e99-4799-94e0-1415cbb122c4","Type":"ContainerStarted","Data":"dffd17473a96b5b5db4333e6a7fde6449ede9dc1a974313c1ee05756bcb21e5f"}
Sep 18 20:21:33.858752 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:33.858698    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nmzkm" podStartSLOduration=1.5989274089999999 podStartE2EDuration="2.858681057s" totalImagesPullingTime="1.259753648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:33.857064016 +0000 UTC m=+62.887990159" watchObservedRunningTime="2026-09-18 20:21:33.858681057 +0000 UTC m=+62.889607196"
Sep 18 20:21:34.798462 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:34.798431    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-phhxn"
Sep 18 20:21:35.677630 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.677595    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8h7pv"]
Sep 18 20:21:35.683169 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.682391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.685711 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.684877    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ssg9g\""
Sep 18 20:21:35.685711 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.685117    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 20:21:35.685711 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.685404    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 20:21:35.685711 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.685580    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 20:21:35.741524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.741087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-tls\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.741524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.741192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b507f6e-4807-4b73-aa93-942579bc41dc-root\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.741524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.741221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.741524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.741248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-wtmp\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.741524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.741289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.741524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.741341    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-textfile\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.741524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.741365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b507f6e-4807-4b73-aa93-942579bc41dc-metrics-client-ca\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.741524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.741389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b507f6e-4807-4b73-aa93-942579bc41dc-sys\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.741524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.741413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7qr6\" (UniqueName: \"kubernetes.io/projected/1b507f6e-4807-4b73-aa93-942579bc41dc-kube-api-access-d7qr6\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-tls\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b507f6e-4807-4b73-aa93-942579bc41dc-root\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-wtmp\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-textfile\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b507f6e-4807-4b73-aa93-942579bc41dc-root\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b507f6e-4807-4b73-aa93-942579bc41dc-metrics-client-ca\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b507f6e-4807-4b73-aa93-942579bc41dc-sys\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.843131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.842725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qr6\" (UniqueName: \"kubernetes.io/projected/1b507f6e-4807-4b73-aa93-942579bc41dc-kube-api-access-d7qr6\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.844030 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:35.843182    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 20:21:35.844030 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:35.843242    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-tls podName:1b507f6e-4807-4b73-aa93-942579bc41dc nodeName:}" failed. No retries permitted until 2026-09-18 20:21:36.343224039 +0000 UTC m=+65.374150157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-tls") pod "node-exporter-8h7pv" (UID: "1b507f6e-4807-4b73-aa93-942579bc41dc") : secret "node-exporter-tls" not found
Sep 18 20:21:35.844030 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.843456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b507f6e-4807-4b73-aa93-942579bc41dc-sys\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.844030 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.843180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-wtmp\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.844030 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.843717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-textfile\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.844030 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.843738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.844315 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.844229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b507f6e-4807-4b73-aa93-942579bc41dc-metrics-client-ca\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.853594 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.848881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:35.853594 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:35.852722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qr6\" (UniqueName: \"kubernetes.io/projected/1b507f6e-4807-4b73-aa93-942579bc41dc-kube-api-access-d7qr6\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:36.349077 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:36.349040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-tls\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:36.353790 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:36.353756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b507f6e-4807-4b73-aa93-942579bc41dc-node-exporter-tls\") pod \"node-exporter-8h7pv\" (UID: \"1b507f6e-4807-4b73-aa93-942579bc41dc\") " pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:36.596304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:36.596261    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8h7pv"
Sep 18 20:21:37.255936 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.255898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:21:37.256406 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.255952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:21:37.258760 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.258739    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 20:21:37.259773 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.259745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 20:21:37.269939 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.269913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a406b73-2c97-49f0-82b9-964afe466106-original-pull-secret\") pod \"global-pull-secret-syncer-msj7n\" (UID: \"1a406b73-2c97-49f0-82b9-964afe466106\") " pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:21:37.270047 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.269954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e160c7e-4ea8-48e5-b932-32be44d1c289-metrics-certs\") pod \"network-metrics-daemon-zfhl9\" (UID: \"4e160c7e-4ea8-48e5-b932-32be44d1c289\") " pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:21:37.357138 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.357101    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:21:37.359499 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.359466    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 20:21:37.368875 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.368851    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msj7n"
Sep 18 20:21:37.369746 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.369722    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 20:21:37.381566 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.381514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ll7\" (UniqueName: \"kubernetes.io/projected/01e6577b-6859-4947-9532-fc0b9edcf3cb-kube-api-access-v2ll7\") pod \"network-check-target-75k5k\" (UID: \"01e6577b-6859-4947-9532-fc0b9edcf3cb\") " pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:21:37.468752 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.468721    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ls8xl\""
Sep 18 20:21:37.478468 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.476728    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfhl9"
Sep 18 20:21:37.667171 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.667065    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5lg4r\""
Sep 18 20:21:37.675110 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:37.675080    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:21:38.058261 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.058227    2566 patch_prober.go:28] interesting pod/image-registry-7cc8799c58-s2j4f container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 20:21:38.058427 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.058282    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f" podUID="1eb0f128-f9ca-49be-ac71-204992e694f4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:21:38.132266 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:38.132098    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b507f6e_4807_4b73_aa93_942579bc41dc.slice/crio-cb5ed35f19b258e86657bb525144ca1a25f81cc7110ea88e4911ce0e42cacb8e WatchSource:0}: Error finding container cb5ed35f19b258e86657bb525144ca1a25f81cc7110ea88e4911ce0e42cacb8e: Status 404 returned error can't find the container with id cb5ed35f19b258e86657bb525144ca1a25f81cc7110ea88e4911ce0e42cacb8e
Sep 18 20:21:38.395076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.394983    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-msj7n"]
Sep 18 20:21:38.409367 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.409328    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfhl9"]
Sep 18 20:21:38.410351 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.410330    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-75k5k"]
Sep 18 20:21:38.478219 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:38.478181    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a406b73_2c97_49f0_82b9_964afe466106.slice/crio-3dc93aad8927f5ecce897847ae53a167d37f0235854f38707fbcf1a2fddad882 WatchSource:0}: Error finding container 3dc93aad8927f5ecce897847ae53a167d37f0235854f38707fbcf1a2fddad882: Status 404 returned error can't find the container with id 3dc93aad8927f5ecce897847ae53a167d37f0235854f38707fbcf1a2fddad882
Sep 18 20:21:38.479209 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:38.479161    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e160c7e_4ea8_48e5_b932_32be44d1c289.slice/crio-93cdbf802859ead477ce6e68d584cd9384a0f5982d2230526c99e463708823ec WatchSource:0}: Error finding container 93cdbf802859ead477ce6e68d584cd9384a0f5982d2230526c99e463708823ec: Status 404 returned error can't find the container with id 93cdbf802859ead477ce6e68d584cd9384a0f5982d2230526c99e463708823ec
Sep 18 20:21:38.480366 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:21:38.480344    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e6577b_6859_4947_9532_fc0b9edcf3cb.slice/crio-cef32ffb4461fba81db8b8f899fd200e6c7d6b734d941eca8afd5c2a9e819cdd WatchSource:0}: Error finding container cef32ffb4461fba81db8b8f899fd200e6c7d6b734d941eca8afd5c2a9e819cdd: Status 404 returned error can't find the container with id cef32ffb4461fba81db8b8f899fd200e6c7d6b734d941eca8afd5c2a9e819cdd
Sep 18 20:21:38.864620 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.864193    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-75k5k" event={"ID":"01e6577b-6859-4947-9532-fc0b9edcf3cb","Type":"ContainerStarted","Data":"cef32ffb4461fba81db8b8f899fd200e6c7d6b734d941eca8afd5c2a9e819cdd"}
Sep 18 20:21:38.866052 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.865889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h7pv" event={"ID":"1b507f6e-4807-4b73-aa93-942579bc41dc","Type":"ContainerStarted","Data":"cb5ed35f19b258e86657bb525144ca1a25f81cc7110ea88e4911ce0e42cacb8e"}
Sep 18 20:21:38.868322 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.868257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4d6c48cb-wswgs" event={"ID":"acfd42f5-7a25-4220-afd6-5162b6b3fb7e","Type":"ContainerStarted","Data":"7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828"}
Sep 18 20:21:38.870873 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.870800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-msj7n" event={"ID":"1a406b73-2c97-49f0-82b9-964afe466106","Type":"ContainerStarted","Data":"3dc93aad8927f5ecce897847ae53a167d37f0235854f38707fbcf1a2fddad882"}
Sep 18 20:21:38.872657 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.872584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfhl9" event={"ID":"4e160c7e-4ea8-48e5-b932-32be44d1c289","Type":"ContainerStarted","Data":"93cdbf802859ead477ce6e68d584cd9384a0f5982d2230526c99e463708823ec"}
Sep 18 20:21:38.887153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:38.887107    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d4d6c48cb-wswgs" podStartSLOduration=1.5101983570000002 podStartE2EDuration="6.88709391s" totalImagesPullingTime="5.376895553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:38.886596983 +0000 UTC m=+67.917523124" watchObservedRunningTime="2026-09-18 20:21:38.88709391 +0000 UTC m=+67.918020050"
Sep 18 20:21:40.459558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:40.459469    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d4d6c48cb-wswgs"]
Sep 18 20:21:41.844419 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.844379    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:21:41.848465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.848436    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:41.851676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 20:21:41.851676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 20:21:41.851676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851370    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 20:21:41.851676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851421    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 20:21:41.851676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851333    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 20:21:41.851676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851486    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 20:21:41.851676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851372    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 20:21:41.851676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851655    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 20:21:41.852144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851687    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2im9jpn1b12l9\""
Sep 18 20:21:41.852144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.851850    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8l4kr\""
Sep 18 20:21:41.852899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.852405    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 20:21:41.852899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.852597    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 20:21:41.852899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.852698    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 20:21:41.852899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.852405    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 20:21:41.853201 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.853050    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 20:21:41.853201 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.853092    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 20:21:41.854265 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.854244    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 20:21:41.864281 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:41.864260    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-config\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006261    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-web-config\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-config-out\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.006934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.007819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96ktc\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-kube-api-access-96ktc\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.007819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.007819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.007819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.006732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.107821 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.107821 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.107821 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-config\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-web-config\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.107997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-config-out\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108827 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108827 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108827 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108827 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96ktc\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-kube-api-access-96ktc\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.108827 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.108524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.109205 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.109179    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.110750 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.109437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.110750 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.109765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.114380 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.112438    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.115050 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.115003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.115707 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.115678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.117106 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.116659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.117106 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.117058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-web-config\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.120095 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.119238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.120289 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.120268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.126253 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.126235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.126813 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.126748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.127156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.127114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-config-out\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.129737 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.128455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.130006 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.129970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-config\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.130128 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.130095    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.130479 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.130457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.131052 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.130988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96ktc\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-kube-api-access-96ktc\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.131283 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.131261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.163061 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.162992    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:21:42.720411 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:42.720375    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:21:48.057132 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:48.057105    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:49.820561 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:49.820468    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b6c9fd4b9-6stdj"
Sep 18 20:21:51.374398 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.374347    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:21:51.935982 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.935940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-5trl7" event={"ID":"2f56239d-1c43-4f8d-905d-7f297ae92eeb","Type":"ContainerStarted","Data":"d148a1465575b9c28620a737b403db9bcbee9c091de92cfc253e861e40c28b57"}
Sep 18 20:21:51.936635 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.936610    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-5trl7"
Sep 18 20:21:51.939972 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.939938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-75k5k" event={"ID":"01e6577b-6859-4947-9532-fc0b9edcf3cb","Type":"ContainerStarted","Data":"9c8aa33c3e0dd9dc77d112a313fe050996a03e46c58d575d82d2bbd580344f8c"}
Sep 18 20:21:51.940267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.940027    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:21:51.942546 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.942456    2566 generic.go:356] "Generic (PLEG): container finished" podID="1b507f6e-4807-4b73-aa93-942579bc41dc" containerID="e91acc2931fd13ea5690f38e9aa63d15274746e33e36079a8a652c730bcc95a5" exitCode=0
Sep 18 20:21:51.942650 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.942547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h7pv" event={"ID":"1b507f6e-4807-4b73-aa93-942579bc41dc","Type":"ContainerDied","Data":"e91acc2931fd13ea5690f38e9aa63d15274746e33e36079a8a652c730bcc95a5"}
Sep 18 20:21:51.945264 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.945231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-msj7n" event={"ID":"1a406b73-2c97-49f0-82b9-964afe466106","Type":"ContainerStarted","Data":"3dfe9f5e337d644dc2f18dcb75af10fef64d83340b7b566c8b7073b6daa06cc3"}
Sep 18 20:21:51.947257 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.947209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerStarted","Data":"87d07f2bc481e19c6a7ac0d76c0d6691f53ec8fdf4fe20fd01cb8b4e99a428ac"}
Sep 18 20:21:51.950635 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.950275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfhl9" event={"ID":"4e160c7e-4ea8-48e5-b932-32be44d1c289","Type":"ContainerStarted","Data":"eeb689406d04a02f0c404efbcac810d8418ad4e4b739c051d280923c70439e51"}
Sep 18 20:21:51.950635 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.950323    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfhl9" event={"ID":"4e160c7e-4ea8-48e5-b932-32be44d1c289","Type":"ContainerStarted","Data":"f01181fd622c5be95a10d026cc84fb3739528996fe759a25523435ca1bc72c41"}
Sep 18 20:21:51.950635 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.950600    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-5trl7"
Sep 18 20:21:51.957494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.957444    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-5trl7" podStartSLOduration=1.500232333 podStartE2EDuration="23.957427492s" totalImagesPullingTime="22.457195159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:51.9552419 +0000 UTC m=+80.986168054" watchObservedRunningTime="2026-09-18 20:21:51.957427492 +0000 UTC m=+80.988353632"
Sep 18 20:21:51.970730 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:51.970660    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-msj7n" podStartSLOduration=68.369832756 podStartE2EDuration="1m20.970453819s" totalImagesPullingTime="12.600621063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:51.969955666 +0000 UTC m=+81.000881807" watchObservedRunningTime="2026-09-18 20:21:51.970453819 +0000 UTC m=+81.001379960"
Sep 18 20:21:52.028662 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:52.028447    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-75k5k" podStartSLOduration=68.208810615 podStartE2EDuration="1m21.028426112s" totalImagesPullingTime="12.819615497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:52.007146781 +0000 UTC m=+81.038072921" watchObservedRunningTime="2026-09-18 20:21:52.028426112 +0000 UTC m=+81.059352253"
Sep 18 20:21:52.049720 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:52.047292    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zfhl9" podStartSLOduration=68.461158791 podStartE2EDuration="1m21.047268498s" totalImagesPullingTime="12.586109707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:52.046281947 +0000 UTC m=+81.077208088" watchObservedRunningTime="2026-09-18 20:21:52.047268498 +0000 UTC m=+81.078194642"
Sep 18 20:21:52.957950 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:52.957914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h7pv" event={"ID":"1b507f6e-4807-4b73-aa93-942579bc41dc","Type":"ContainerStarted","Data":"b57803fce198ae512fe797333e8a5b8ec92bd6086bdf429328cce59e269d1a56"}
Sep 18 20:21:52.958374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:52.957977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h7pv" event={"ID":"1b507f6e-4807-4b73-aa93-942579bc41dc","Type":"ContainerStarted","Data":"6d04acd9b3fe83577910ca50cf1859a25531824a63979ea83a4151b24d6f6816"}
Sep 18 20:21:52.959741 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:52.959713    2566 generic.go:356] "Generic (PLEG): container finished" podID="64ef49b8-5069-439d-90e5-b6df1964226d" containerID="40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc" exitCode=0
Sep 18 20:21:52.959864 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:52.959746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerDied","Data":"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"}
Sep 18 20:21:52.984602 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:52.984555    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8h7pv" podStartSLOduration=4.965825888 podStartE2EDuration="17.984539479s" totalImagesPullingTime="12.949588851s" totalInitContainerRuntime="69.12474ms" isStatefulPod=true podCreationTimestamp="2026-09-18 20:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:52.984117556 +0000 UTC m=+82.015043700" watchObservedRunningTime="2026-09-18 20:21:52.984539479 +0000 UTC m=+82.015465670"
Sep 18 20:21:53.072416 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.072305    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f" podUID="1eb0f128-f9ca-49be-ac71-204992e694f4" containerName="registry" containerID="cri-o://ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62" gracePeriod=30
Sep 18 20:21:53.371087 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.371034    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:53.518101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.518061    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-certificates\") pod \"1eb0f128-f9ca-49be-ac71-204992e694f4\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") "
Sep 18 20:21:53.518101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.518137    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtqlz\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-kube-api-access-xtqlz\") pod \"1eb0f128-f9ca-49be-ac71-204992e694f4\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") "
Sep 18 20:21:53.518101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.518160    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-bound-sa-token\") pod \"1eb0f128-f9ca-49be-ac71-204992e694f4\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") "
Sep 18 20:21:53.518951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.518196    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-installation-pull-secrets\") pod \"1eb0f128-f9ca-49be-ac71-204992e694f4\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") "
Sep 18 20:21:53.518951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.518252    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-image-registry-private-configuration\") pod \"1eb0f128-f9ca-49be-ac71-204992e694f4\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") "
Sep 18 20:21:53.518951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.518288    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1eb0f128-f9ca-49be-ac71-204992e694f4-ca-trust-extracted\") pod \"1eb0f128-f9ca-49be-ac71-204992e694f4\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") "
Sep 18 20:21:53.518951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.518314    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") pod \"1eb0f128-f9ca-49be-ac71-204992e694f4\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") "
Sep 18 20:21:53.518951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.518342    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-trusted-ca\") pod \"1eb0f128-f9ca-49be-ac71-204992e694f4\" (UID: \"1eb0f128-f9ca-49be-ac71-204992e694f4\") "
Sep 18 20:21:53.518951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.518643    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1eb0f128-f9ca-49be-ac71-204992e694f4" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:53.520064 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.520004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1eb0f128-f9ca-49be-ac71-204992e694f4" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:53.521978 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.521788    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1eb0f128-f9ca-49be-ac71-204992e694f4" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:53.523279 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.523246    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1eb0f128-f9ca-49be-ac71-204992e694f4" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:53.523722 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.523585    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-kube-api-access-xtqlz" (OuterVolumeSpecName: "kube-api-access-xtqlz") pod "1eb0f128-f9ca-49be-ac71-204992e694f4" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4"). InnerVolumeSpecName "kube-api-access-xtqlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:53.523991 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.523955    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1eb0f128-f9ca-49be-ac71-204992e694f4" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:53.524710 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.524634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1eb0f128-f9ca-49be-ac71-204992e694f4" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:53.531890 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.531859    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb0f128-f9ca-49be-ac71-204992e694f4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1eb0f128-f9ca-49be-ac71-204992e694f4" (UID: "1eb0f128-f9ca-49be-ac71-204992e694f4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:53.619064 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.618953    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-installation-pull-secrets\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.619064 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.618988    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1eb0f128-f9ca-49be-ac71-204992e694f4-image-registry-private-configuration\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.619064 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.619003    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1eb0f128-f9ca-49be-ac71-204992e694f4-ca-trust-extracted\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.619064 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.619038    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.619064 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.619052    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-trusted-ca\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.619064 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.619064    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1eb0f128-f9ca-49be-ac71-204992e694f4-registry-certificates\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.619463 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.619077    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtqlz\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-kube-api-access-xtqlz\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.619463 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.619087    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb0f128-f9ca-49be-ac71-204992e694f4-bound-sa-token\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:21:53.966976 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.966817    2566 generic.go:356] "Generic (PLEG): container finished" podID="1eb0f128-f9ca-49be-ac71-204992e694f4" containerID="ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62" exitCode=0
Sep 18 20:21:53.967740 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.967705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f" event={"ID":"1eb0f128-f9ca-49be-ac71-204992e694f4","Type":"ContainerDied","Data":"ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62"}
Sep 18 20:21:53.967869 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.967764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f" event={"ID":"1eb0f128-f9ca-49be-ac71-204992e694f4","Type":"ContainerDied","Data":"6fe353813e7eb8a6e0ca11708380bf841399f0b2e39bc70c868ab73c952e9eac"}
Sep 18 20:21:53.967869 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.967792    2566 scope.go:117] "RemoveContainer" containerID="ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62"
Sep 18 20:21:53.967869 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.967832    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cc8799c58-s2j4f"
Sep 18 20:21:53.979941 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.979870    2566 scope.go:117] "RemoveContainer" containerID="ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62"
Sep 18 20:21:53.980517 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:21:53.980485    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62\": container with ID starting with ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62 not found: ID does not exist" containerID="ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62"
Sep 18 20:21:53.980635 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.980521    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62"} err="failed to get container status \"ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62\": rpc error: code = NotFound desc = could not find container \"ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62\": container with ID starting with ad005be7319e30c6f7ca82960c23ac221e8a105521e22584f7f1004a3920bd62 not found: ID does not exist"
Sep 18 20:21:53.997100 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:53.997071    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cc8799c58-s2j4f"]
Sep 18 20:21:54.001784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:54.001693    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cc8799c58-s2j4f"]
Sep 18 20:21:55.461662 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:55.461622    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb0f128-f9ca-49be-ac71-204992e694f4" path="/var/lib/kubelet/pods/1eb0f128-f9ca-49be-ac71-204992e694f4/volumes"
Sep 18 20:21:56.986372 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:56.986334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerStarted","Data":"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"}
Sep 18 20:21:56.987055 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:21:56.986967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerStarted","Data":"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"}
Sep 18 20:22:00.005198 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:00.005129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerStarted","Data":"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"}
Sep 18 20:22:00.005602 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:00.005211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerStarted","Data":"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"}
Sep 18 20:22:00.005602 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:00.005254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerStarted","Data":"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"}
Sep 18 20:22:00.005602 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:00.005295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerStarted","Data":"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"}
Sep 18 20:22:00.035508 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:00.035424    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=11.368240853 podStartE2EDuration="19.035401747s" totalImagesPullingTime="7.642455496s" totalInitContainerRuntime="24.705398ms" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:00.033821318 +0000 UTC m=+89.064747493" watchObservedRunningTime="2026-09-18 20:22:00.035401747 +0000 UTC m=+89.066327890"
Sep 18 20:22:02.164052 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:02.164000    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:22:05.914967 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:05.914926    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d4d6c48cb-wswgs" podUID="acfd42f5-7a25-4220-afd6-5162b6b3fb7e" containerName="console" containerID="cri-o://7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828" gracePeriod=15
Sep 18 20:22:06.184714 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.184660    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:22:06.219725 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.219702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9fzz\" (UniqueName: \"kubernetes.io/projected/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-kube-api-access-m9fzz\") pod \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") "
Sep 18 20:22:06.219849 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.219740    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-oauth-serving-cert\") pod \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") "
Sep 18 20:22:06.219849 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.219785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-serving-cert\") pod \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") "
Sep 18 20:22:06.219849 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.219812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-oauth-config\") pod \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") "
Sep 18 20:22:06.219849 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.219840    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-service-ca\") pod \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") "
Sep 18 20:22:06.220078 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.219976    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-config\") pod \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\" (UID: \"acfd42f5-7a25-4220-afd6-5162b6b3fb7e\") "
Sep 18 20:22:06.220338 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.220297    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "acfd42f5-7a25-4220-afd6-5162b6b3fb7e" (UID: "acfd42f5-7a25-4220-afd6-5162b6b3fb7e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:06.220456 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.220365    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-service-ca" (OuterVolumeSpecName: "service-ca") pod "acfd42f5-7a25-4220-afd6-5162b6b3fb7e" (UID: "acfd42f5-7a25-4220-afd6-5162b6b3fb7e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:06.220456 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.220443    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-config" (OuterVolumeSpecName: "console-config") pod "acfd42f5-7a25-4220-afd6-5162b6b3fb7e" (UID: "acfd42f5-7a25-4220-afd6-5162b6b3fb7e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:06.222153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.222122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-kube-api-access-m9fzz" (OuterVolumeSpecName: "kube-api-access-m9fzz") pod "acfd42f5-7a25-4220-afd6-5162b6b3fb7e" (UID: "acfd42f5-7a25-4220-afd6-5162b6b3fb7e"). InnerVolumeSpecName "kube-api-access-m9fzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:06.222251 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.222132    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "acfd42f5-7a25-4220-afd6-5162b6b3fb7e" (UID: "acfd42f5-7a25-4220-afd6-5162b6b3fb7e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:06.222251 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.222180    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "acfd42f5-7a25-4220-afd6-5162b6b3fb7e" (UID: "acfd42f5-7a25-4220-afd6-5162b6b3fb7e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:06.320562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.320534    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:22:06.320562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.320558    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fzz\" (UniqueName: \"kubernetes.io/projected/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-kube-api-access-m9fzz\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:22:06.320693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.320569    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-oauth-serving-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:22:06.320693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.320577    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-serving-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:22:06.320693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.320586    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-console-oauth-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:22:06.320693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:06.320594    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acfd42f5-7a25-4220-afd6-5162b6b3fb7e-service-ca\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:22:07.033103 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.033070    2566 generic.go:356] "Generic (PLEG): container finished" podID="acfd42f5-7a25-4220-afd6-5162b6b3fb7e" containerID="7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828" exitCode=0
Sep 18 20:22:07.033512 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.033114    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d4d6c48cb-wswgs"
Sep 18 20:22:07.033512 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.033143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4d6c48cb-wswgs" event={"ID":"acfd42f5-7a25-4220-afd6-5162b6b3fb7e","Type":"ContainerDied","Data":"7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828"}
Sep 18 20:22:07.033512 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.033182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d4d6c48cb-wswgs" event={"ID":"acfd42f5-7a25-4220-afd6-5162b6b3fb7e","Type":"ContainerDied","Data":"9c02bd25994554ea9639501c0818eb494fc61e51ec9b3c5efc0048e422db0076"}
Sep 18 20:22:07.033512 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.033202    2566 scope.go:117] "RemoveContainer" containerID="7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828"
Sep 18 20:22:07.042110 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.042004    2566 scope.go:117] "RemoveContainer" containerID="7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828"
Sep 18 20:22:07.042349 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:22:07.042332    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828\": container with ID starting with 7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828 not found: ID does not exist" containerID="7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828"
Sep 18 20:22:07.042391 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.042358    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828"} err="failed to get container status \"7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828\": rpc error: code = NotFound desc = could not find container \"7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828\": container with ID starting with 7453cec2acaf9a91f683b7247d31b66413cf075dbe2653ce6c0108c71c7d7828 not found: ID does not exist"
Sep 18 20:22:07.060757 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.060732    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d4d6c48cb-wswgs"]
Sep 18 20:22:07.068460 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.068441    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d4d6c48cb-wswgs"]
Sep 18 20:22:07.460306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:07.460235    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acfd42f5-7a25-4220-afd6-5162b6b3fb7e" path="/var/lib/kubelet/pods/acfd42f5-7a25-4220-afd6-5162b6b3fb7e/volumes"
Sep 18 20:22:22.963145 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:22.963115    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-75k5k"
Sep 18 20:22:42.163322 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:42.163291    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:22:42.182324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:42.182298    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:22:43.177744 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:22:43.177719    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:00.238813 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.238780    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:00.239520 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.239451    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="prometheus" containerID="cri-o://7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c" gracePeriod=600
Sep 18 20:23:00.239520 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.239515    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a" gracePeriod=600
Sep 18 20:23:00.239740 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.239561    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy-web" containerID="cri-o://66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83" gracePeriod=600
Sep 18 20:23:00.239740 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.239477    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy" containerID="cri-o://c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59" gracePeriod=600
Sep 18 20:23:00.239740 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.239499    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="config-reloader" containerID="cri-o://d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98" gracePeriod=600
Sep 18 20:23:00.239897 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.239768    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="thanos-sidecar" containerID="cri-o://5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd" gracePeriod=600
Sep 18 20:23:00.488320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.488297    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:00.627002 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.626970    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-kube-rbac-proxy\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627002 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627024    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-trusted-ca-bundle\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627270 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627051    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-grpc-tls\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627270 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-web-config\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627270 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627192    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-thanos-prometheus-http-client-file\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627270 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627245    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-db\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627277    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627326    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-metrics-client-certs\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627367    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-config\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627391    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-1\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-0\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627431    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:00.627486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-metrics-client-ca\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627517    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-serving-certs-ca-bundle\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627545    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627575    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-kubelet-serving-ca-bundle\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627605    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-config-out\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627629    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96ktc\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-kube-api-access-96ktc\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627661    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-tls\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627686    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-tls-assets\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.627846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627719    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-2\") pod \"64ef49b8-5069-439d-90e5-b6df1964226d\" (UID: \"64ef49b8-5069-439d-90e5-b6df1964226d\") "
Sep 18 20:23:00.628278 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.627970    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.628278 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.628222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:00.629467 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.628556    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:00.629467 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.629342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:00.629709 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.629526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:00.629709 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.629564    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:00.630473 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.630446    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:00.630864 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.630816    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:00.631544 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.631499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-config" (OuterVolumeSpecName: "config") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:00.631719 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.631694    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:00.632072 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.632044    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:00.632258 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.632169    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:00.632258 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.632241    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:00.632395 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.632300    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:00.632572 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.632540    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:00.632675 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.632554    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:00.632749 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.632714    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-config-out" (OuterVolumeSpecName: "config-out") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:00.633154 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.633134    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:00.633227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.633190    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-kube-api-access-96ktc" (OuterVolumeSpecName: "kube-api-access-96ktc") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "kube-api-access-96ktc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:00.641233 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.641213    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-web-config" (OuterVolumeSpecName: "web-config") pod "64ef49b8-5069-439d-90e5-b6df1964226d" (UID: "64ef49b8-5069-439d-90e5-b6df1964226d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:00.728942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.728907    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.728942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.728942    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-db\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.728942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.728952    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.728942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.728961    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-metrics-client-certs\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.728972    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.728984    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.728994    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729003    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-metrics-client-ca\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729026    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729037    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729045    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729053    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64ef49b8-5069-439d-90e5-b6df1964226d-config-out\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729061    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96ktc\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-kube-api-access-96ktc\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729069    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729076    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64ef49b8-5069-439d-90e5-b6df1964226d-tls-assets\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729084    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/64ef49b8-5069-439d-90e5-b6df1964226d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729092    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-kube-rbac-proxy\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729106    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-secret-grpc-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:00.729194 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:00.729115    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64ef49b8-5069-439d-90e5-b6df1964226d-web-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:23:01.226739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226702    2566 generic.go:356] "Generic (PLEG): container finished" podID="64ef49b8-5069-439d-90e5-b6df1964226d" containerID="575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a" exitCode=0
Sep 18 20:23:01.226739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226726    2566 generic.go:356] "Generic (PLEG): container finished" podID="64ef49b8-5069-439d-90e5-b6df1964226d" containerID="c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59" exitCode=0
Sep 18 20:23:01.226739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226732    2566 generic.go:356] "Generic (PLEG): container finished" podID="64ef49b8-5069-439d-90e5-b6df1964226d" containerID="66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83" exitCode=0
Sep 18 20:23:01.226739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226737    2566 generic.go:356] "Generic (PLEG): container finished" podID="64ef49b8-5069-439d-90e5-b6df1964226d" containerID="5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd" exitCode=0
Sep 18 20:23:01.226739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226743    2566 generic.go:356] "Generic (PLEG): container finished" podID="64ef49b8-5069-439d-90e5-b6df1964226d" containerID="d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98" exitCode=0
Sep 18 20:23:01.226739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226748    2566 generic.go:356] "Generic (PLEG): container finished" podID="64ef49b8-5069-439d-90e5-b6df1964226d" containerID="7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c" exitCode=0
Sep 18 20:23:01.227076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226772    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.227076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerDied","Data":"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"}
Sep 18 20:23:01.227076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerDied","Data":"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"}
Sep 18 20:23:01.227076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerDied","Data":"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"}
Sep 18 20:23:01.227076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226918    2566 scope.go:117] "RemoveContainer" containerID="575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"
Sep 18 20:23:01.227076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.226918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerDied","Data":"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"}
Sep 18 20:23:01.227076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.227047    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerDied","Data":"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"}
Sep 18 20:23:01.227076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.227078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerDied","Data":"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"}
Sep 18 20:23:01.227404 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.227119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64ef49b8-5069-439d-90e5-b6df1964226d","Type":"ContainerDied","Data":"87d07f2bc481e19c6a7ac0d76c0d6691f53ec8fdf4fe20fd01cb8b4e99a428ac"}
Sep 18 20:23:01.234564 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.234544    2566 scope.go:117] "RemoveContainer" containerID="c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"
Sep 18 20:23:01.241581 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.241409    2566 scope.go:117] "RemoveContainer" containerID="66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"
Sep 18 20:23:01.247768 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.247751    2566 scope.go:117] "RemoveContainer" containerID="5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"
Sep 18 20:23:01.251521 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.251497    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:01.254255 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.254235    2566 scope.go:117] "RemoveContainer" containerID="d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"
Sep 18 20:23:01.257622 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.257596    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:01.268162 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.268140    2566 scope.go:117] "RemoveContainer" containerID="7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"
Sep 18 20:23:01.274960 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.274944    2566 scope.go:117] "RemoveContainer" containerID="40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"
Sep 18 20:23:01.281323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.281307    2566 scope.go:117] "RemoveContainer" containerID="575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"
Sep 18 20:23:01.281603 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:23:01.281585    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": container with ID starting with 575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a not found: ID does not exist" containerID="575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"
Sep 18 20:23:01.281668 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.281610    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"} err="failed to get container status \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": rpc error: code = NotFound desc = could not find container \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": container with ID starting with 575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a not found: ID does not exist"
Sep 18 20:23:01.281668 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.281629    2566 scope.go:117] "RemoveContainer" containerID="c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"
Sep 18 20:23:01.281864 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:23:01.281847    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": container with ID starting with c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59 not found: ID does not exist" containerID="c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"
Sep 18 20:23:01.281908 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.281871    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"} err="failed to get container status \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": rpc error: code = NotFound desc = could not find container \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": container with ID starting with c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59 not found: ID does not exist"
Sep 18 20:23:01.281908 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.281887    2566 scope.go:117] "RemoveContainer" containerID="66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"
Sep 18 20:23:01.282122 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:23:01.282107    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": container with ID starting with 66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83 not found: ID does not exist" containerID="66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"
Sep 18 20:23:01.282168 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.282126    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"} err="failed to get container status \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": rpc error: code = NotFound desc = could not find container \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": container with ID starting with 66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83 not found: ID does not exist"
Sep 18 20:23:01.282168 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.282139    2566 scope.go:117] "RemoveContainer" containerID="5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"
Sep 18 20:23:01.282343 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:23:01.282326    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": container with ID starting with 5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd not found: ID does not exist" containerID="5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"
Sep 18 20:23:01.282395 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.282345    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"} err="failed to get container status \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": rpc error: code = NotFound desc = could not find container \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": container with ID starting with 5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd not found: ID does not exist"
Sep 18 20:23:01.282395 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.282360    2566 scope.go:117] "RemoveContainer" containerID="d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"
Sep 18 20:23:01.282609 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:23:01.282594    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": container with ID starting with d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98 not found: ID does not exist" containerID="d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"
Sep 18 20:23:01.282645 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.282614    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"} err="failed to get container status \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": rpc error: code = NotFound desc = could not find container \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": container with ID starting with d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98 not found: ID does not exist"
Sep 18 20:23:01.282645 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.282628    2566 scope.go:117] "RemoveContainer" containerID="7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"
Sep 18 20:23:01.282825 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:23:01.282812    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": container with ID starting with 7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c not found: ID does not exist" containerID="7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"
Sep 18 20:23:01.282863 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.282827    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"} err="failed to get container status \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": rpc error: code = NotFound desc = could not find container \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": container with ID starting with 7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c not found: ID does not exist"
Sep 18 20:23:01.282863 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.282838    2566 scope.go:117] "RemoveContainer" containerID="40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"
Sep 18 20:23:01.283071 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:23:01.283051    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": container with ID starting with 40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc not found: ID does not exist" containerID="40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"
Sep 18 20:23:01.283115 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.283079    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"} err="failed to get container status \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": rpc error: code = NotFound desc = could not find container \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": container with ID starting with 40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc not found: ID does not exist"
Sep 18 20:23:01.283115 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.283099    2566 scope.go:117] "RemoveContainer" containerID="575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"
Sep 18 20:23:01.283358 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.283340    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"} err="failed to get container status \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": rpc error: code = NotFound desc = could not find container \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": container with ID starting with 575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a not found: ID does not exist"
Sep 18 20:23:01.283426 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.283359    2566 scope.go:117] "RemoveContainer" containerID="c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"
Sep 18 20:23:01.283598 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.283583    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"} err="failed to get container status \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": rpc error: code = NotFound desc = could not find container \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": container with ID starting with c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59 not found: ID does not exist"
Sep 18 20:23:01.283649 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.283597    2566 scope.go:117] "RemoveContainer" containerID="66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"
Sep 18 20:23:01.283833 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.283814    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"} err="failed to get container status \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": rpc error: code = NotFound desc = could not find container \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": container with ID starting with 66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83 not found: ID does not exist"
Sep 18 20:23:01.283886 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.283834    2566 scope.go:117] "RemoveContainer" containerID="5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"
Sep 18 20:23:01.284103 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284052    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"} err="failed to get container status \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": rpc error: code = NotFound desc = could not find container \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": container with ID starting with 5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd not found: ID does not exist"
Sep 18 20:23:01.284103 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284066    2566 scope.go:117] "RemoveContainer" containerID="d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"
Sep 18 20:23:01.284278 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284265    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"} err="failed to get container status \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": rpc error: code = NotFound desc = could not find container \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": container with ID starting with d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98 not found: ID does not exist"
Sep 18 20:23:01.284278 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284277    2566 scope.go:117] "RemoveContainer" containerID="7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"
Sep 18 20:23:01.284494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284475    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"} err="failed to get container status \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": rpc error: code = NotFound desc = could not find container \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": container with ID starting with 7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c not found: ID does not exist"
Sep 18 20:23:01.284494 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284496    2566 scope.go:117] "RemoveContainer" containerID="40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"
Sep 18 20:23:01.284731 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284716    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"} err="failed to get container status \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": rpc error: code = NotFound desc = could not find container \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": container with ID starting with 40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc not found: ID does not exist"
Sep 18 20:23:01.284774 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284731    2566 scope.go:117] "RemoveContainer" containerID="575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"
Sep 18 20:23:01.284949 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284931    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"} err="failed to get container status \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": rpc error: code = NotFound desc = could not find container \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": container with ID starting with 575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a not found: ID does not exist"
Sep 18 20:23:01.284949 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.284949    2566 scope.go:117] "RemoveContainer" containerID="c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"
Sep 18 20:23:01.285174 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.285159    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"} err="failed to get container status \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": rpc error: code = NotFound desc = could not find container \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": container with ID starting with c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59 not found: ID does not exist"
Sep 18 20:23:01.285213 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.285174    2566 scope.go:117] "RemoveContainer" containerID="66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"
Sep 18 20:23:01.285392 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.285374    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"} err="failed to get container status \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": rpc error: code = NotFound desc = could not find container \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": container with ID starting with 66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83 not found: ID does not exist"
Sep 18 20:23:01.285446 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.285392    2566 scope.go:117] "RemoveContainer" containerID="5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"
Sep 18 20:23:01.285604 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.285587    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"} err="failed to get container status \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": rpc error: code = NotFound desc = could not find container \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": container with ID starting with 5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd not found: ID does not exist"
Sep 18 20:23:01.285604 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.285603    2566 scope.go:117] "RemoveContainer" containerID="d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"
Sep 18 20:23:01.285817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.285800    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"} err="failed to get container status \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": rpc error: code = NotFound desc = could not find container \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": container with ID starting with d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98 not found: ID does not exist"
Sep 18 20:23:01.285859 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.285818    2566 scope.go:117] "RemoveContainer" containerID="7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"
Sep 18 20:23:01.286064 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.286046    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"} err="failed to get container status \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": rpc error: code = NotFound desc = could not find container \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": container with ID starting with 7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c not found: ID does not exist"
Sep 18 20:23:01.286122 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.286065    2566 scope.go:117] "RemoveContainer" containerID="40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"
Sep 18 20:23:01.286277 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.286261    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"} err="failed to get container status \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": rpc error: code = NotFound desc = could not find container \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": container with ID starting with 40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc not found: ID does not exist"
Sep 18 20:23:01.286319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.286277    2566 scope.go:117] "RemoveContainer" containerID="575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"
Sep 18 20:23:01.286487 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.286473    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"} err="failed to get container status \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": rpc error: code = NotFound desc = could not find container \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": container with ID starting with 575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a not found: ID does not exist"
Sep 18 20:23:01.286487 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.286485    2566 scope.go:117] "RemoveContainer" containerID="c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"
Sep 18 20:23:01.286718 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.286697    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"} err="failed to get container status \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": rpc error: code = NotFound desc = could not find container \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": container with ID starting with c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59 not found: ID does not exist"
Sep 18 20:23:01.286718 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.286716    2566 scope.go:117] "RemoveContainer" containerID="66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"
Sep 18 20:23:01.287167 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.287141    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"} err="failed to get container status \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": rpc error: code = NotFound desc = could not find container \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": container with ID starting with 66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83 not found: ID does not exist"
Sep 18 20:23:01.287256 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.287168    2566 scope.go:117] "RemoveContainer" containerID="5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"
Sep 18 20:23:01.287562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.287527    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"} err="failed to get container status \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": rpc error: code = NotFound desc = could not find container \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": container with ID starting with 5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd not found: ID does not exist"
Sep 18 20:23:01.287562 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.287549    2566 scope.go:117] "RemoveContainer" containerID="d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"
Sep 18 20:23:01.287830 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.287811    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"} err="failed to get container status \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": rpc error: code = NotFound desc = could not find container \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": container with ID starting with d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98 not found: ID does not exist"
Sep 18 20:23:01.287830 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.287830    2566 scope.go:117] "RemoveContainer" containerID="7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"
Sep 18 20:23:01.288090 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288071    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"} err="failed to get container status \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": rpc error: code = NotFound desc = could not find container \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": container with ID starting with 7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c not found: ID does not exist"
Sep 18 20:23:01.288090 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288091    2566 scope.go:117] "RemoveContainer" containerID="40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"
Sep 18 20:23:01.288362 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288344    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"} err="failed to get container status \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": rpc error: code = NotFound desc = could not find container \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": container with ID starting with 40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc not found: ID does not exist"
Sep 18 20:23:01.288407 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288362    2566 scope.go:117] "RemoveContainer" containerID="575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"
Sep 18 20:23:01.288533 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288518    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:01.288592 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288574    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"} err="failed to get container status \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": rpc error: code = NotFound desc = could not find container \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": container with ID starting with 575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a not found: ID does not exist"
Sep 18 20:23:01.288671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288593    2566 scope.go:117] "RemoveContainer" containerID="c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"
Sep 18 20:23:01.288763 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288751    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="thanos-sidecar"
Sep 18 20:23:01.288808 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288765    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="thanos-sidecar"
Sep 18 20:23:01.288808 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288772    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy-web"
Sep 18 20:23:01.288808 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288781    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy-web"
Sep 18 20:23:01.288808 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288792    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy-thanos"
Sep 18 20:23:01.288808 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288798    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy-thanos"
Sep 18 20:23:01.288808 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288795    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"} err="failed to get container status \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": rpc error: code = NotFound desc = could not find container \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": container with ID starting with c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59 not found: ID does not exist"
Sep 18 20:23:01.288808 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288805    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb0f128-f9ca-49be-ac71-204992e694f4" containerName="registry"
Sep 18 20:23:01.288808 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288811    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb0f128-f9ca-49be-ac71-204992e694f4" containerName="registry"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288830    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="init-config-reloader"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288835    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="init-config-reloader"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288841    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="prometheus"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288846    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="prometheus"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288852    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acfd42f5-7a25-4220-afd6-5162b6b3fb7e" containerName="console"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288857    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="acfd42f5-7a25-4220-afd6-5162b6b3fb7e" containerName="console"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288863    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288868    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288876    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="config-reloader"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288881    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="config-reloader"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288810    2566 scope.go:117] "RemoveContainer" containerID="66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288920    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="config-reloader"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288926    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="thanos-sidecar"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288932    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb0f128-f9ca-49be-ac71-204992e694f4" containerName="registry"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288938    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="acfd42f5-7a25-4220-afd6-5162b6b3fb7e" containerName="console"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288943    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy-web"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288948    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288953    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="kube-rbac-proxy-thanos"
Sep 18 20:23:01.289104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.288960    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" containerName="prometheus"
Sep 18 20:23:01.289739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.289119    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"} err="failed to get container status \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": rpc error: code = NotFound desc = could not find container \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": container with ID starting with 66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83 not found: ID does not exist"
Sep 18 20:23:01.289739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.289134    2566 scope.go:117] "RemoveContainer" containerID="5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"
Sep 18 20:23:01.289739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.289368    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"} err="failed to get container status \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": rpc error: code = NotFound desc = could not find container \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": container with ID starting with 5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd not found: ID does not exist"
Sep 18 20:23:01.289739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.289384    2566 scope.go:117] "RemoveContainer" containerID="d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"
Sep 18 20:23:01.289739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.289634    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"} err="failed to get container status \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": rpc error: code = NotFound desc = could not find container \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": container with ID starting with d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98 not found: ID does not exist"
Sep 18 20:23:01.289739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.289652    2566 scope.go:117] "RemoveContainer" containerID="7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"
Sep 18 20:23:01.289946 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.289898    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"} err="failed to get container status \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": rpc error: code = NotFound desc = could not find container \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": container with ID starting with 7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c not found: ID does not exist"
Sep 18 20:23:01.289946 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.289915    2566 scope.go:117] "RemoveContainer" containerID="40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"
Sep 18 20:23:01.290175 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.290154    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"} err="failed to get container status \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": rpc error: code = NotFound desc = could not find container \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": container with ID starting with 40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc not found: ID does not exist"
Sep 18 20:23:01.290252 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.290176    2566 scope.go:117] "RemoveContainer" containerID="575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"
Sep 18 20:23:01.290445 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.290429    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a"} err="failed to get container status \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": rpc error: code = NotFound desc = could not find container \"575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a\": container with ID starting with 575eb254c3dab3c0decb05dee31ca90cdf4ac9a4ec64638a66d983c69ccc0c7a not found: ID does not exist"
Sep 18 20:23:01.290510 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.290444    2566 scope.go:117] "RemoveContainer" containerID="c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"
Sep 18 20:23:01.290694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.290674    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59"} err="failed to get container status \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": rpc error: code = NotFound desc = could not find container \"c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59\": container with ID starting with c06062c84f7365b96b840530036483fb7a976f6d9a71c4cc63fab2c11a759b59 not found: ID does not exist"
Sep 18 20:23:01.290694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.290693    2566 scope.go:117] "RemoveContainer" containerID="66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"
Sep 18 20:23:01.290951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.290935    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83"} err="failed to get container status \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": rpc error: code = NotFound desc = could not find container \"66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83\": container with ID starting with 66289ff47a58c213a650461386f79d8fe2c7aa6942d11ce29a28cd3f044c2b83 not found: ID does not exist"
Sep 18 20:23:01.290997 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.290952    2566 scope.go:117] "RemoveContainer" containerID="5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"
Sep 18 20:23:01.291219 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.291202    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd"} err="failed to get container status \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": rpc error: code = NotFound desc = could not find container \"5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd\": container with ID starting with 5c9b0347d8bbcede594709d167eeaae2043c09b6689f825c88a52e81ced200dd not found: ID does not exist"
Sep 18 20:23:01.291294 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.291220    2566 scope.go:117] "RemoveContainer" containerID="d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"
Sep 18 20:23:01.291476 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.291460    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98"} err="failed to get container status \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": rpc error: code = NotFound desc = could not find container \"d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98\": container with ID starting with d3563fd3b7101ee45204c1332fe7c2a82204479e73a8f8b97e61dc8384c8df98 not found: ID does not exist"
Sep 18 20:23:01.291518 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.291486    2566 scope.go:117] "RemoveContainer" containerID="7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"
Sep 18 20:23:01.291715 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.291694    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c"} err="failed to get container status \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": rpc error: code = NotFound desc = could not find container \"7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c\": container with ID starting with 7837cda89209e76a0b5440b0a20fbbf9259c4193226263316715b7918624d06c not found: ID does not exist"
Sep 18 20:23:01.291715 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.291714    2566 scope.go:117] "RemoveContainer" containerID="40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"
Sep 18 20:23:01.291958 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.291940    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc"} err="failed to get container status \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": rpc error: code = NotFound desc = could not find container \"40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc\": container with ID starting with 40734d4de4792a5a4a355e3c4063a97c7c952f7667d9e760f5e8b846e7ca66fc not found: ID does not exist"
Sep 18 20:23:01.292441 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.292426    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.294986 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.294968    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8l4kr\""
Sep 18 20:23:01.295141 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.294987    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 20:23:01.295226 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.295211    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 20:23:01.295288 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.295246    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 20:23:01.295336 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.295291    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 20:23:01.296069 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.295831    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 20:23:01.302955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.296958    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 20:23:01.302955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.297927    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 20:23:01.302955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.300939    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 20:23:01.302955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.301241    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 20:23:01.302955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.301479    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 20:23:01.302955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.301681    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 20:23:01.302955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.302230    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 20:23:01.302955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.302471    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2im9jpn1b12l9\""
Sep 18 20:23:01.304154 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.304133    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 20:23:01.305962 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.305943    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 20:23:01.306487 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.306473    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 20:23:01.314519 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.314494    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:01.433471 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433653 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433653 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433653 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433653 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433653 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433896 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-config\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433896 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfphp\" (UniqueName: \"kubernetes.io/projected/38a506e5-80c7-4c05-8345-68a0cad94e5a-kube-api-access-dfphp\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433896 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433896 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433896 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433896 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-web-config\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.433896 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.434208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.434208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.434208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.434208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.433995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.434208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.434032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.434208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.434050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/38a506e5-80c7-4c05-8345-68a0cad94e5a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.434208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.434065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/38a506e5-80c7-4c05-8345-68a0cad94e5a-config-out\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.458700 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.458671    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ef49b8-5069-439d-90e5-b6df1964226d" path="/var/lib/kubelet/pods/64ef49b8-5069-439d-90e5-b6df1964226d/volumes"
Sep 18 20:23:01.534697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.534811 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.534811 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.534811 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/38a506e5-80c7-4c05-8345-68a0cad94e5a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.534811 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/38a506e5-80c7-4c05-8345-68a0cad94e5a-config-out\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.534977    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-config\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfphp\" (UniqueName: \"kubernetes.io/projected/38a506e5-80c7-4c05-8345-68a0cad94e5a-kube-api-access-dfphp\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-web-config\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.535693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535544    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.536830 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.535895    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.536830 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.536103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.536999 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.536907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.537086 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.537004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.537222 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.537194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.537731 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.537708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/38a506e5-80c7-4c05-8345-68a0cad94e5a-config-out\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.538051 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.538006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.538698 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.538673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.539425 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.539381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-web-config\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.539569 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.539546    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/38a506e5-80c7-4c05-8345-68a0cad94e5a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.539728 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.539590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.539880 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.539852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.540206 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.540187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/38a506e5-80c7-4c05-8345-68a0cad94e5a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.540394 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.540365    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.540827 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.540791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.540827 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.540815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.541180 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.541154    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.541180 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.541168    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/38a506e5-80c7-4c05-8345-68a0cad94e5a-config\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.542936 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.542916    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfphp\" (UniqueName: \"kubernetes.io/projected/38a506e5-80c7-4c05-8345-68a0cad94e5a-kube-api-access-dfphp\") pod \"prometheus-k8s-0\" (UID: \"38a506e5-80c7-4c05-8345-68a0cad94e5a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.605387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.605360    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:23:01.885348 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:01.885323    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 20:23:01.887254 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:23:01.887228    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a506e5_80c7_4c05_8345_68a0cad94e5a.slice/crio-ddfb1e8ee7c2d48809a2a22e76b17ef49246ed6e15b42a079f3ce44e4b6096ef WatchSource:0}: Error finding container ddfb1e8ee7c2d48809a2a22e76b17ef49246ed6e15b42a079f3ce44e4b6096ef: Status 404 returned error can't find the container with id ddfb1e8ee7c2d48809a2a22e76b17ef49246ed6e15b42a079f3ce44e4b6096ef
Sep 18 20:23:02.231851 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:02.231750    2566 generic.go:356] "Generic (PLEG): container finished" podID="38a506e5-80c7-4c05-8345-68a0cad94e5a" containerID="35a34429cdd36e33c8a74503e7d61a1345e1a3f612876a329dfe8589b4206ff4" exitCode=0
Sep 18 20:23:02.231851 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:02.231830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"38a506e5-80c7-4c05-8345-68a0cad94e5a","Type":"ContainerDied","Data":"35a34429cdd36e33c8a74503e7d61a1345e1a3f612876a329dfe8589b4206ff4"}
Sep 18 20:23:02.232071 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:02.231896    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"38a506e5-80c7-4c05-8345-68a0cad94e5a","Type":"ContainerStarted","Data":"ddfb1e8ee7c2d48809a2a22e76b17ef49246ed6e15b42a079f3ce44e4b6096ef"}
Sep 18 20:23:03.239693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:03.239659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"38a506e5-80c7-4c05-8345-68a0cad94e5a","Type":"ContainerStarted","Data":"67c72b17333266f8bcb47f5c7563daa5c90168a4f0cba90d0c8842da848345fd"}
Sep 18 20:23:03.240087 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:03.239755    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"38a506e5-80c7-4c05-8345-68a0cad94e5a","Type":"ContainerStarted","Data":"0df13934ae96fc46b2ce2541de531debddd52e80bfac970a9f3b5f5ac7d2c4be"}
Sep 18 20:23:03.240087 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:03.239792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"38a506e5-80c7-4c05-8345-68a0cad94e5a","Type":"ContainerStarted","Data":"4d655d80c87c5e4caf526374b9a459ed9ffab4d7654483dc286fba20f634c2c1"}
Sep 18 20:23:03.240087 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:03.239819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"38a506e5-80c7-4c05-8345-68a0cad94e5a","Type":"ContainerStarted","Data":"435e288bc9a7a806d4a96b5be66db93ef91afcee0adb2c7b15da490f33860e3a"}
Sep 18 20:23:03.240087 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:03.239842    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"38a506e5-80c7-4c05-8345-68a0cad94e5a","Type":"ContainerStarted","Data":"fc4165c81bffde34768a406c2b6126c1e281da47ec1fde83192176461b3c3594"}
Sep 18 20:23:03.240087 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:03.239884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"38a506e5-80c7-4c05-8345-68a0cad94e5a","Type":"ContainerStarted","Data":"213db3afe6813fb8385088e80d34f704bb4f78ddb3890df491a008e8dcf4912d"}
Sep 18 20:23:03.268902 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:03.268825    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.251011609 podStartE2EDuration="2.268809764s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.798155ms" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:03.268560831 +0000 UTC m=+152.299486970" watchObservedRunningTime="2026-09-18 20:23:03.268809764 +0000 UTC m=+152.299735904"
Sep 18 20:23:06.606457 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:23:06.606423    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:24:01.605696 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:24:01.605664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:24:01.620772 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:24:01.620750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:24:02.462813 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:24:02.462785    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 20:25:31.336737 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:25:31.336707    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:25:31.343942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:25:31.343914    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:25:31.349867 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:25:31.349850    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 20:27:33.303535 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.303502    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-vm9vw"]
Sep 18 20:27:33.305681 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.305663    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:33.308408 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.308376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-xrt9l\""
Sep 18 20:27:33.308627 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.308414    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 20:27:33.309335 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.309315    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 20:27:33.309512 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.309495    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 18 20:27:33.324740 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.324716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vm9vw"]
Sep 18 20:27:33.325632 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.325613    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-4vrqz"]
Sep 18 20:27:33.327389 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.327375    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.330354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.330299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 20:27:33.330658 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.330638    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-sx9r8\""
Sep 18 20:27:33.338027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.337996    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 20:27:33.357773 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.357740    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-4vrqz"]
Sep 18 20:27:33.465561 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.465526    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czsds\" (UniqueName: \"kubernetes.io/projected/83271296-0142-4900-86a5-ff5b7020ff8e-kube-api-access-czsds\") pod \"model-serving-api-7b954ffc95-vm9vw\" (UID: \"83271296-0142-4900-86a5-ff5b7020ff8e\") " pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:33.465561 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.465559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/012fc48d-f413-4e1f-90d5-341604a5854a-cert\") pod \"odh-model-controller-7b966d7984-4vrqz\" (UID: \"012fc48d-f413-4e1f-90d5-341604a5854a\") " pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.465817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.465576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/012fc48d-f413-4e1f-90d5-341604a5854a-metrics-certs\") pod \"odh-model-controller-7b966d7984-4vrqz\" (UID: \"012fc48d-f413-4e1f-90d5-341604a5854a\") " pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.465817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.465603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbd8\" (UniqueName: \"kubernetes.io/projected/012fc48d-f413-4e1f-90d5-341604a5854a-kube-api-access-zlbd8\") pod \"odh-model-controller-7b966d7984-4vrqz\" (UID: \"012fc48d-f413-4e1f-90d5-341604a5854a\") " pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.465817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.465682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83271296-0142-4900-86a5-ff5b7020ff8e-tls-certs\") pod \"model-serving-api-7b954ffc95-vm9vw\" (UID: \"83271296-0142-4900-86a5-ff5b7020ff8e\") " pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:33.566701 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.566627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czsds\" (UniqueName: \"kubernetes.io/projected/83271296-0142-4900-86a5-ff5b7020ff8e-kube-api-access-czsds\") pod \"model-serving-api-7b954ffc95-vm9vw\" (UID: \"83271296-0142-4900-86a5-ff5b7020ff8e\") " pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:33.566701 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.566663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/012fc48d-f413-4e1f-90d5-341604a5854a-cert\") pod \"odh-model-controller-7b966d7984-4vrqz\" (UID: \"012fc48d-f413-4e1f-90d5-341604a5854a\") " pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.566701 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.566679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/012fc48d-f413-4e1f-90d5-341604a5854a-metrics-certs\") pod \"odh-model-controller-7b966d7984-4vrqz\" (UID: \"012fc48d-f413-4e1f-90d5-341604a5854a\") " pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.566923 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.566793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbd8\" (UniqueName: \"kubernetes.io/projected/012fc48d-f413-4e1f-90d5-341604a5854a-kube-api-access-zlbd8\") pod \"odh-model-controller-7b966d7984-4vrqz\" (UID: \"012fc48d-f413-4e1f-90d5-341604a5854a\") " pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.566923 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.566859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83271296-0142-4900-86a5-ff5b7020ff8e-tls-certs\") pod \"model-serving-api-7b954ffc95-vm9vw\" (UID: \"83271296-0142-4900-86a5-ff5b7020ff8e\") " pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:33.569085 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.569053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/012fc48d-f413-4e1f-90d5-341604a5854a-metrics-certs\") pod \"odh-model-controller-7b966d7984-4vrqz\" (UID: \"012fc48d-f413-4e1f-90d5-341604a5854a\") " pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.569211 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.569140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/012fc48d-f413-4e1f-90d5-341604a5854a-cert\") pod \"odh-model-controller-7b966d7984-4vrqz\" (UID: \"012fc48d-f413-4e1f-90d5-341604a5854a\") " pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.569260 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.569229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83271296-0142-4900-86a5-ff5b7020ff8e-tls-certs\") pod \"model-serving-api-7b954ffc95-vm9vw\" (UID: \"83271296-0142-4900-86a5-ff5b7020ff8e\") " pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:33.575060 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.575037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbd8\" (UniqueName: \"kubernetes.io/projected/012fc48d-f413-4e1f-90d5-341604a5854a-kube-api-access-zlbd8\") pod \"odh-model-controller-7b966d7984-4vrqz\" (UID: \"012fc48d-f413-4e1f-90d5-341604a5854a\") " pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.575452 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.575435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czsds\" (UniqueName: \"kubernetes.io/projected/83271296-0142-4900-86a5-ff5b7020ff8e-kube-api-access-czsds\") pod \"model-serving-api-7b954ffc95-vm9vw\" (UID: \"83271296-0142-4900-86a5-ff5b7020ff8e\") " pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:33.616463 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.616442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:33.640227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.640206    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:33.882937 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.882884    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vm9vw"]
Sep 18 20:27:33.885354 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:27:33.885323    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83271296_0142_4900_86a5_ff5b7020ff8e.slice/crio-1512a5b4ac71bb0a4cddf8b2855e3265651b7fb0300c7244ba23826f21f4815f WatchSource:0}: Error finding container 1512a5b4ac71bb0a4cddf8b2855e3265651b7fb0300c7244ba23826f21f4815f: Status 404 returned error can't find the container with id 1512a5b4ac71bb0a4cddf8b2855e3265651b7fb0300c7244ba23826f21f4815f
Sep 18 20:27:33.887104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.887089    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:27:33.913449 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:33.913424    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-4vrqz"]
Sep 18 20:27:33.916685 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:27:33.916662    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012fc48d_f413_4e1f_90d5_341604a5854a.slice/crio-3d5896104b54f8b0d9c8bc6403ceecb2df9ffc35f3902bb47cd673137559cb17 WatchSource:0}: Error finding container 3d5896104b54f8b0d9c8bc6403ceecb2df9ffc35f3902bb47cd673137559cb17: Status 404 returned error can't find the container with id 3d5896104b54f8b0d9c8bc6403ceecb2df9ffc35f3902bb47cd673137559cb17
Sep 18 20:27:34.173870 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:34.173782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-4vrqz" event={"ID":"012fc48d-f413-4e1f-90d5-341604a5854a","Type":"ContainerStarted","Data":"3d5896104b54f8b0d9c8bc6403ceecb2df9ffc35f3902bb47cd673137559cb17"}
Sep 18 20:27:34.174850 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:34.174827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vm9vw" event={"ID":"83271296-0142-4900-86a5-ff5b7020ff8e","Type":"ContainerStarted","Data":"1512a5b4ac71bb0a4cddf8b2855e3265651b7fb0300c7244ba23826f21f4815f"}
Sep 18 20:27:38.194457 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:38.194421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vm9vw" event={"ID":"83271296-0142-4900-86a5-ff5b7020ff8e","Type":"ContainerStarted","Data":"c96af3e5e8678927f04296bda03e285760ab4a1571be3f3a990eade09a56a641"}
Sep 18 20:27:38.194882 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:38.194595    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:38.195950 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:38.195927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-4vrqz" event={"ID":"012fc48d-f413-4e1f-90d5-341604a5854a","Type":"ContainerStarted","Data":"fd874e718ed86e8de78cde1b843d522cbea78a73a7942d385109bf1548ab4510"}
Sep 18 20:27:38.196067 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:38.196038    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:38.213182 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:38.213140    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-vm9vw" podStartSLOduration=1.728137627 podStartE2EDuration="5.213123331s" totalImagesPullingTime="3.484985704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:38.21216085 +0000 UTC m=+427.243086991" watchObservedRunningTime="2026-09-18 20:27:38.213123331 +0000 UTC m=+427.244049472"
Sep 18 20:27:38.229144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:38.229098    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-4vrqz" podStartSLOduration=1.731254276 podStartE2EDuration="5.229084902s" totalImagesPullingTime="3.497830626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:38.228446597 +0000 UTC m=+427.259372737" watchObservedRunningTime="2026-09-18 20:27:38.229084902 +0000 UTC m=+427.260011056"
Sep 18 20:27:49.201423 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:49.201393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-4vrqz"
Sep 18 20:27:49.203221 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:49.203194    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-vm9vw"
Sep 18 20:27:50.032858 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.032825    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-6lpqh"]
Sep 18 20:27:50.038578 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.038558    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6lpqh"
Sep 18 20:27:50.041190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.041171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 20:27:50.041309 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.041192    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wb7jw\""
Sep 18 20:27:50.041309 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.041208    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 20:27:50.044603 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.044581    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6lpqh"]
Sep 18 20:27:50.199288 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.199251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k4qp\" (UniqueName: \"kubernetes.io/projected/b668aec6-09f6-4a9a-98ff-76536f37b802-kube-api-access-7k4qp\") pod \"s3-init-6lpqh\" (UID: \"b668aec6-09f6-4a9a-98ff-76536f37b802\") " pod="kserve/s3-init-6lpqh"
Sep 18 20:27:50.199455 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.199305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b668aec6-09f6-4a9a-98ff-76536f37b802-model-cache\") pod \"s3-init-6lpqh\" (UID: \"b668aec6-09f6-4a9a-98ff-76536f37b802\") " pod="kserve/s3-init-6lpqh"
Sep 18 20:27:50.300264 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.300191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b668aec6-09f6-4a9a-98ff-76536f37b802-model-cache\") pod \"s3-init-6lpqh\" (UID: \"b668aec6-09f6-4a9a-98ff-76536f37b802\") " pod="kserve/s3-init-6lpqh"
Sep 18 20:27:50.300264 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.300252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k4qp\" (UniqueName: \"kubernetes.io/projected/b668aec6-09f6-4a9a-98ff-76536f37b802-kube-api-access-7k4qp\") pod \"s3-init-6lpqh\" (UID: \"b668aec6-09f6-4a9a-98ff-76536f37b802\") " pod="kserve/s3-init-6lpqh"
Sep 18 20:27:50.300624 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.300538    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b668aec6-09f6-4a9a-98ff-76536f37b802-model-cache\") pod \"s3-init-6lpqh\" (UID: \"b668aec6-09f6-4a9a-98ff-76536f37b802\") " pod="kserve/s3-init-6lpqh"
Sep 18 20:27:50.308324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.308306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k4qp\" (UniqueName: \"kubernetes.io/projected/b668aec6-09f6-4a9a-98ff-76536f37b802-kube-api-access-7k4qp\") pod \"s3-init-6lpqh\" (UID: \"b668aec6-09f6-4a9a-98ff-76536f37b802\") " pod="kserve/s3-init-6lpqh"
Sep 18 20:27:50.367920 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.367880    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6lpqh"
Sep 18 20:27:50.636607 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:50.636530    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6lpqh"]
Sep 18 20:27:50.639591 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:27:50.639562    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb668aec6_09f6_4a9a_98ff_76536f37b802.slice/crio-5a70c667eabd2fc3fa776e38262cb8c2a2705a28eeee4ed14d1909e50e2962a2 WatchSource:0}: Error finding container 5a70c667eabd2fc3fa776e38262cb8c2a2705a28eeee4ed14d1909e50e2962a2: Status 404 returned error can't find the container with id 5a70c667eabd2fc3fa776e38262cb8c2a2705a28eeee4ed14d1909e50e2962a2
Sep 18 20:27:51.244164 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:51.244129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6lpqh" event={"ID":"b668aec6-09f6-4a9a-98ff-76536f37b802","Type":"ContainerStarted","Data":"5a70c667eabd2fc3fa776e38262cb8c2a2705a28eeee4ed14d1909e50e2962a2"}
Sep 18 20:27:54.260157 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:54.260116    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6lpqh" event={"ID":"b668aec6-09f6-4a9a-98ff-76536f37b802","Type":"ContainerStarted","Data":"0c5a30e924933dd69c21cdf35d6f41802f2a8a2870e19ea56880e875c94ec12e"}
Sep 18 20:27:56.271942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:56.271912    2566 generic.go:356] "Generic (PLEG): container finished" podID="b668aec6-09f6-4a9a-98ff-76536f37b802" containerID="0c5a30e924933dd69c21cdf35d6f41802f2a8a2870e19ea56880e875c94ec12e" exitCode=0
Sep 18 20:27:56.272326 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:27:56.271980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6lpqh" event={"ID":"b668aec6-09f6-4a9a-98ff-76536f37b802","Type":"ContainerDied","Data":"0c5a30e924933dd69c21cdf35d6f41802f2a8a2870e19ea56880e875c94ec12e"}
Sep 18 20:28:01.296810 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:28:01.296773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6lpqh" event={"ID":"b668aec6-09f6-4a9a-98ff-76536f37b802","Type":"ContainerStarted","Data":"f8216d8e96dbc6750d5a4c144db56e82d128c03a8d7c0c13f92e31d8442bafdf"}
Sep 18 20:28:01.316845 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:28:01.316793    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-6lpqh" podStartSLOduration=2.136860504 podStartE2EDuration="11.316778875s" totalImagesPullingTime="7.34565731s" totalInitContainerRuntime="1.834261061s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:01.315693034 +0000 UTC m=+450.346619175" watchObservedRunningTime="2026-09-18 20:28:01.316778875 +0000 UTC m=+450.347705016"
Sep 18 20:29:05.520081 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:05.520050    2566 generic.go:356] "Generic (PLEG): container finished" podID="b668aec6-09f6-4a9a-98ff-76536f37b802" containerID="f8216d8e96dbc6750d5a4c144db56e82d128c03a8d7c0c13f92e31d8442bafdf" exitCode=0
Sep 18 20:29:05.520440 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:05.520096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6lpqh" event={"ID":"b668aec6-09f6-4a9a-98ff-76536f37b802","Type":"ContainerDied","Data":"f8216d8e96dbc6750d5a4c144db56e82d128c03a8d7c0c13f92e31d8442bafdf"}
Sep 18 20:29:06.645859 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:06.645837    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6lpqh"
Sep 18 20:29:06.769256 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:06.769217    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k4qp\" (UniqueName: \"kubernetes.io/projected/b668aec6-09f6-4a9a-98ff-76536f37b802-kube-api-access-7k4qp\") pod \"b668aec6-09f6-4a9a-98ff-76536f37b802\" (UID: \"b668aec6-09f6-4a9a-98ff-76536f37b802\") "
Sep 18 20:29:06.769431 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:06.769274    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b668aec6-09f6-4a9a-98ff-76536f37b802-model-cache\") pod \"b668aec6-09f6-4a9a-98ff-76536f37b802\" (UID: \"b668aec6-09f6-4a9a-98ff-76536f37b802\") "
Sep 18 20:29:06.771392 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:06.771356    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b668aec6-09f6-4a9a-98ff-76536f37b802-kube-api-access-7k4qp" (OuterVolumeSpecName: "kube-api-access-7k4qp") pod "b668aec6-09f6-4a9a-98ff-76536f37b802" (UID: "b668aec6-09f6-4a9a-98ff-76536f37b802"). InnerVolumeSpecName "kube-api-access-7k4qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:29:06.787858 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:06.787834    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b668aec6-09f6-4a9a-98ff-76536f37b802-model-cache" (OuterVolumeSpecName: "model-cache") pod "b668aec6-09f6-4a9a-98ff-76536f37b802" (UID: "b668aec6-09f6-4a9a-98ff-76536f37b802"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:06.870235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:06.870210    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k4qp\" (UniqueName: \"kubernetes.io/projected/b668aec6-09f6-4a9a-98ff-76536f37b802-kube-api-access-7k4qp\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:29:06.870235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:06.870231    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b668aec6-09f6-4a9a-98ff-76536f37b802-model-cache\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:29:07.528197 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:07.528170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6lpqh" event={"ID":"b668aec6-09f6-4a9a-98ff-76536f37b802","Type":"ContainerDied","Data":"5a70c667eabd2fc3fa776e38262cb8c2a2705a28eeee4ed14d1909e50e2962a2"}
Sep 18 20:29:07.528197 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:07.528199    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6lpqh"
Sep 18 20:29:07.528197 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:07.528207    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a70c667eabd2fc3fa776e38262cb8c2a2705a28eeee4ed14d1909e50e2962a2"
Sep 18 20:29:08.239979 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.239950    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"]
Sep 18 20:29:08.240331 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.240246    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b668aec6-09f6-4a9a-98ff-76536f37b802" containerName="s3-init"
Sep 18 20:29:08.240331 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.240269    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b668aec6-09f6-4a9a-98ff-76536f37b802" containerName="s3-init"
Sep 18 20:29:08.240331 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.240285    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b668aec6-09f6-4a9a-98ff-76536f37b802" containerName="download-hf-model"
Sep 18 20:29:08.240331 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.240294    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b668aec6-09f6-4a9a-98ff-76536f37b802" containerName="download-hf-model"
Sep 18 20:29:08.240458 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.240349    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b668aec6-09f6-4a9a-98ff-76536f37b802" containerName="s3-init"
Sep 18 20:29:08.243197 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.243182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:08.245720 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.245695    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 20:29:08.245830 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.245757    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wb7jw\""
Sep 18 20:29:08.251784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.251761    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"]
Sep 18 20:29:08.381545 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.381516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65dbd63f-c16b-4a66-8b68-37bd173954fb-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-kwfg6\" (UID: \"65dbd63f-c16b-4a66-8b68-37bd173954fb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:08.381695 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.381572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vfq\" (UniqueName: \"kubernetes.io/projected/65dbd63f-c16b-4a66-8b68-37bd173954fb-kube-api-access-m2vfq\") pod \"seaweedfs-tls-custom-6644bf5d6c-kwfg6\" (UID: \"65dbd63f-c16b-4a66-8b68-37bd173954fb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:08.482069 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.482037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65dbd63f-c16b-4a66-8b68-37bd173954fb-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-kwfg6\" (UID: \"65dbd63f-c16b-4a66-8b68-37bd173954fb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:08.482230 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.482120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vfq\" (UniqueName: \"kubernetes.io/projected/65dbd63f-c16b-4a66-8b68-37bd173954fb-kube-api-access-m2vfq\") pod \"seaweedfs-tls-custom-6644bf5d6c-kwfg6\" (UID: \"65dbd63f-c16b-4a66-8b68-37bd173954fb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:08.482375 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.482345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65dbd63f-c16b-4a66-8b68-37bd173954fb-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-kwfg6\" (UID: \"65dbd63f-c16b-4a66-8b68-37bd173954fb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:08.493948 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.493883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vfq\" (UniqueName: \"kubernetes.io/projected/65dbd63f-c16b-4a66-8b68-37bd173954fb-kube-api-access-m2vfq\") pod \"seaweedfs-tls-custom-6644bf5d6c-kwfg6\" (UID: \"65dbd63f-c16b-4a66-8b68-37bd173954fb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:08.551908 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.551884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:08.819800 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:08.819774    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"]
Sep 18 20:29:08.821964 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:29:08.821938    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dbd63f_c16b_4a66_8b68_37bd173954fb.slice/crio-83d2ace4d0940983f319431a092eb15a0f86c0a5cbecad6f4e2d209a6ce8b712 WatchSource:0}: Error finding container 83d2ace4d0940983f319431a092eb15a0f86c0a5cbecad6f4e2d209a6ce8b712: Status 404 returned error can't find the container with id 83d2ace4d0940983f319431a092eb15a0f86c0a5cbecad6f4e2d209a6ce8b712
Sep 18 20:29:09.536533 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:09.536498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6" event={"ID":"65dbd63f-c16b-4a66-8b68-37bd173954fb","Type":"ContainerStarted","Data":"83d2ace4d0940983f319431a092eb15a0f86c0a5cbecad6f4e2d209a6ce8b712"}
Sep 18 20:29:11.545296 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:11.545274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6" event={"ID":"65dbd63f-c16b-4a66-8b68-37bd173954fb","Type":"ContainerStarted","Data":"3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a"}
Sep 18 20:29:12.563696 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:12.563641    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6" podStartSLOduration=1.9022812820000001 podStartE2EDuration="4.563620246s" totalImagesPullingTime="2.661338964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:12.563371252 +0000 UTC m=+521.594297397" watchObservedRunningTime="2026-09-18 20:29:12.563620246 +0000 UTC m=+521.594546385"
Sep 18 20:29:13.614921 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:13.614882    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"]
Sep 18 20:29:14.555085 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:14.555046    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6" podUID="65dbd63f-c16b-4a66-8b68-37bd173954fb" containerName="seaweedfs-tls-custom" containerID="cri-o://3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a" gracePeriod=30
Sep 18 20:29:16.183793 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.183770    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:16.236867 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.236832    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2vfq\" (UniqueName: \"kubernetes.io/projected/65dbd63f-c16b-4a66-8b68-37bd173954fb-kube-api-access-m2vfq\") pod \"65dbd63f-c16b-4a66-8b68-37bd173954fb\" (UID: \"65dbd63f-c16b-4a66-8b68-37bd173954fb\") "
Sep 18 20:29:16.237005 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.236883    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65dbd63f-c16b-4a66-8b68-37bd173954fb-data\") pod \"65dbd63f-c16b-4a66-8b68-37bd173954fb\" (UID: \"65dbd63f-c16b-4a66-8b68-37bd173954fb\") "
Sep 18 20:29:16.238188 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.238160    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dbd63f-c16b-4a66-8b68-37bd173954fb-data" (OuterVolumeSpecName: "data") pod "65dbd63f-c16b-4a66-8b68-37bd173954fb" (UID: "65dbd63f-c16b-4a66-8b68-37bd173954fb"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:16.238884 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.238854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dbd63f-c16b-4a66-8b68-37bd173954fb-kube-api-access-m2vfq" (OuterVolumeSpecName: "kube-api-access-m2vfq") pod "65dbd63f-c16b-4a66-8b68-37bd173954fb" (UID: "65dbd63f-c16b-4a66-8b68-37bd173954fb"). InnerVolumeSpecName "kube-api-access-m2vfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:29:16.338346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.338277    2566 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65dbd63f-c16b-4a66-8b68-37bd173954fb-data\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:29:16.338346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.338303    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2vfq\" (UniqueName: \"kubernetes.io/projected/65dbd63f-c16b-4a66-8b68-37bd173954fb-kube-api-access-m2vfq\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:29:16.563572 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.563539    2566 generic.go:356] "Generic (PLEG): container finished" podID="65dbd63f-c16b-4a66-8b68-37bd173954fb" containerID="3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a" exitCode=0
Sep 18 20:29:16.563743 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.563588    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"
Sep 18 20:29:16.563743 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.563626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6" event={"ID":"65dbd63f-c16b-4a66-8b68-37bd173954fb","Type":"ContainerDied","Data":"3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a"}
Sep 18 20:29:16.563743 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.563673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6" event={"ID":"65dbd63f-c16b-4a66-8b68-37bd173954fb","Type":"ContainerDied","Data":"83d2ace4d0940983f319431a092eb15a0f86c0a5cbecad6f4e2d209a6ce8b712"}
Sep 18 20:29:16.563743 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.563694    2566 scope.go:117] "RemoveContainer" containerID="3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a"
Sep 18 20:29:16.573359 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.573343    2566 scope.go:117] "RemoveContainer" containerID="3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a"
Sep 18 20:29:16.573646 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:29:16.573629    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a\": container with ID starting with 3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a not found: ID does not exist" containerID="3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a"
Sep 18 20:29:16.573697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.573655    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a"} err="failed to get container status \"3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a\": rpc error: code = NotFound desc = could not find container \"3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a\": container with ID starting with 3e0329d8d4fa8afec0e662004f316fed2564620c32bfb4c257344265067fca1a not found: ID does not exist"
Sep 18 20:29:16.584607 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.584584    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"]
Sep 18 20:29:16.587918 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.587898    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-kwfg6"]
Sep 18 20:29:16.614383 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.614326    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"]
Sep 18 20:29:16.614654 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.614630    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65dbd63f-c16b-4a66-8b68-37bd173954fb" containerName="seaweedfs-tls-custom"
Sep 18 20:29:16.614654 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.614651    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65dbd63f-c16b-4a66-8b68-37bd173954fb" containerName="seaweedfs-tls-custom"
Sep 18 20:29:16.614772 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.614761    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="65dbd63f-c16b-4a66-8b68-37bd173954fb" containerName="seaweedfs-tls-custom"
Sep 18 20:29:16.618892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.618876    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.621667 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.621649    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 20:29:16.621667 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.621661    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wb7jw\""
Sep 18 20:29:16.621802 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.621674    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 18 20:29:16.626379 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.626358    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"]
Sep 18 20:29:16.641532 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.641513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/d0ce9076-49b4-4fda-a945-4c409458f061-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-2zv9m\" (UID: \"d0ce9076-49b4-4fda-a945-4c409458f061\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.641619 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.641547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d0ce9076-49b4-4fda-a945-4c409458f061-data\") pod \"seaweedfs-tls-custom-8b44d767d-2zv9m\" (UID: \"d0ce9076-49b4-4fda-a945-4c409458f061\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.641619 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.641614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdw5f\" (UniqueName: \"kubernetes.io/projected/d0ce9076-49b4-4fda-a945-4c409458f061-kube-api-access-gdw5f\") pod \"seaweedfs-tls-custom-8b44d767d-2zv9m\" (UID: \"d0ce9076-49b4-4fda-a945-4c409458f061\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.742038 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.741998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d0ce9076-49b4-4fda-a945-4c409458f061-data\") pod \"seaweedfs-tls-custom-8b44d767d-2zv9m\" (UID: \"d0ce9076-49b4-4fda-a945-4c409458f061\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.742156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.742102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdw5f\" (UniqueName: \"kubernetes.io/projected/d0ce9076-49b4-4fda-a945-4c409458f061-kube-api-access-gdw5f\") pod \"seaweedfs-tls-custom-8b44d767d-2zv9m\" (UID: \"d0ce9076-49b4-4fda-a945-4c409458f061\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.742156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.742129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/d0ce9076-49b4-4fda-a945-4c409458f061-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-2zv9m\" (UID: \"d0ce9076-49b4-4fda-a945-4c409458f061\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.742410 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.742389    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d0ce9076-49b4-4fda-a945-4c409458f061-data\") pod \"seaweedfs-tls-custom-8b44d767d-2zv9m\" (UID: \"d0ce9076-49b4-4fda-a945-4c409458f061\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.744534 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.744517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/d0ce9076-49b4-4fda-a945-4c409458f061-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-2zv9m\" (UID: \"d0ce9076-49b4-4fda-a945-4c409458f061\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.750886 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.750868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdw5f\" (UniqueName: \"kubernetes.io/projected/d0ce9076-49b4-4fda-a945-4c409458f061-kube-api-access-gdw5f\") pod \"seaweedfs-tls-custom-8b44d767d-2zv9m\" (UID: \"d0ce9076-49b4-4fda-a945-4c409458f061\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:16.928150 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:16.928069    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"
Sep 18 20:29:17.195681 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.195606    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-2zv9m"]
Sep 18 20:29:17.199397 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:29:17.199373    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ce9076_49b4_4fda_a945_4c409458f061.slice/crio-2600bc880705fe1fc07b7f7f9b9166a31ab0068350b0129eb99f619460ff04e1 WatchSource:0}: Error finding container 2600bc880705fe1fc07b7f7f9b9166a31ab0068350b0129eb99f619460ff04e1: Status 404 returned error can't find the container with id 2600bc880705fe1fc07b7f7f9b9166a31ab0068350b0129eb99f619460ff04e1
Sep 18 20:29:17.459903 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.459875    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65dbd63f-c16b-4a66-8b68-37bd173954fb" path="/var/lib/kubelet/pods/65dbd63f-c16b-4a66-8b68-37bd173954fb/volumes"
Sep 18 20:29:17.569720 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.569685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m" event={"ID":"d0ce9076-49b4-4fda-a945-4c409458f061","Type":"ContainerStarted","Data":"fa1428b4a522d0ecddd20d140b47d8c621b2f3fe3656e3ed3f856bd4dff60ca0"}
Sep 18 20:29:17.569870 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.569738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m" event={"ID":"d0ce9076-49b4-4fda-a945-4c409458f061","Type":"ContainerStarted","Data":"2600bc880705fe1fc07b7f7f9b9166a31ab0068350b0129eb99f619460ff04e1"}
Sep 18 20:29:17.584472 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.584430    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-2zv9m" podStartSLOduration=1.33604269 podStartE2EDuration="1.584416325s" totalImagesPullingTime="248.373635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:17.583704608 +0000 UTC m=+526.614630747" watchObservedRunningTime="2026-09-18 20:29:17.584416325 +0000 UTC m=+526.615342464"
Sep 18 20:29:17.864576 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.864542    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-5dcw2"]
Sep 18 20:29:17.867580 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.867564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:17.869846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.869824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 20:29:17.875283 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.875260    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-5dcw2"]
Sep 18 20:29:17.952584 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.952555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrkm\" (UniqueName: \"kubernetes.io/projected/092b6d59-d454-41e9-9f87-703efabe4cd0-kube-api-access-clrkm\") pod \"s3-tls-init-custom-5dcw2\" (UID: \"092b6d59-d454-41e9-9f87-703efabe4cd0\") " pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:17.952701 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:17.952672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b6d59-d454-41e9-9f87-703efabe4cd0-model-cache\") pod \"s3-tls-init-custom-5dcw2\" (UID: \"092b6d59-d454-41e9-9f87-703efabe4cd0\") " pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:18.053766 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:18.053735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b6d59-d454-41e9-9f87-703efabe4cd0-model-cache\") pod \"s3-tls-init-custom-5dcw2\" (UID: \"092b6d59-d454-41e9-9f87-703efabe4cd0\") " pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:18.053929 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:18.053797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clrkm\" (UniqueName: \"kubernetes.io/projected/092b6d59-d454-41e9-9f87-703efabe4cd0-kube-api-access-clrkm\") pod \"s3-tls-init-custom-5dcw2\" (UID: \"092b6d59-d454-41e9-9f87-703efabe4cd0\") " pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:18.054160 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:18.054137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b6d59-d454-41e9-9f87-703efabe4cd0-model-cache\") pod \"s3-tls-init-custom-5dcw2\" (UID: \"092b6d59-d454-41e9-9f87-703efabe4cd0\") " pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:18.062636 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:18.062618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrkm\" (UniqueName: \"kubernetes.io/projected/092b6d59-d454-41e9-9f87-703efabe4cd0-kube-api-access-clrkm\") pod \"s3-tls-init-custom-5dcw2\" (UID: \"092b6d59-d454-41e9-9f87-703efabe4cd0\") " pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:18.193387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:18.193307    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:18.464941 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:18.464920    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-5dcw2"]
Sep 18 20:29:18.467253 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:29:18.467228    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod092b6d59_d454_41e9_9f87_703efabe4cd0.slice/crio-d33f0c3d4106d143f0dde02ca2a9ed588f733784b4c80c38ca0f1bde2f4bc226 WatchSource:0}: Error finding container d33f0c3d4106d143f0dde02ca2a9ed588f733784b4c80c38ca0f1bde2f4bc226: Status 404 returned error can't find the container with id d33f0c3d4106d143f0dde02ca2a9ed588f733784b4c80c38ca0f1bde2f4bc226
Sep 18 20:29:18.574493 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:18.574457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5dcw2" event={"ID":"092b6d59-d454-41e9-9f87-703efabe4cd0","Type":"ContainerStarted","Data":"efa22d0fdfe9845489919b6c186f52871adab0566e48826bbbc4223e85989d20"}
Sep 18 20:29:18.574617 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:18.574500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5dcw2" event={"ID":"092b6d59-d454-41e9-9f87-703efabe4cd0","Type":"ContainerStarted","Data":"d33f0c3d4106d143f0dde02ca2a9ed588f733784b4c80c38ca0f1bde2f4bc226"}
Sep 18 20:29:20.583555 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:20.583524    2566 generic.go:356] "Generic (PLEG): container finished" podID="092b6d59-d454-41e9-9f87-703efabe4cd0" containerID="efa22d0fdfe9845489919b6c186f52871adab0566e48826bbbc4223e85989d20" exitCode=0
Sep 18 20:29:20.583919 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:20.583601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5dcw2" event={"ID":"092b6d59-d454-41e9-9f87-703efabe4cd0","Type":"ContainerDied","Data":"efa22d0fdfe9845489919b6c186f52871adab0566e48826bbbc4223e85989d20"}
Sep 18 20:29:21.588829 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:21.588793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5dcw2" event={"ID":"092b6d59-d454-41e9-9f87-703efabe4cd0","Type":"ContainerStarted","Data":"5a09304670966b06582e0695e252aecf121797e4dcd58945572a3391fe7261ae"}
Sep 18 20:29:21.605443 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:21.605398    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-5dcw2" podStartSLOduration=2.737710186 podStartE2EDuration="4.605383622s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.867673436s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:21.604888378 +0000 UTC m=+530.635814518" watchObservedRunningTime="2026-09-18 20:29:21.605383622 +0000 UTC m=+530.636309763"
Sep 18 20:29:27.616702 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:27.616667    2566 generic.go:356] "Generic (PLEG): container finished" podID="092b6d59-d454-41e9-9f87-703efabe4cd0" containerID="5a09304670966b06582e0695e252aecf121797e4dcd58945572a3391fe7261ae" exitCode=0
Sep 18 20:29:27.617188 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:27.616739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5dcw2" event={"ID":"092b6d59-d454-41e9-9f87-703efabe4cd0","Type":"ContainerDied","Data":"5a09304670966b06582e0695e252aecf121797e4dcd58945572a3391fe7261ae"}
Sep 18 20:29:28.750680 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:28.750659    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:28.842364 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:28.842344    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clrkm\" (UniqueName: \"kubernetes.io/projected/092b6d59-d454-41e9-9f87-703efabe4cd0-kube-api-access-clrkm\") pod \"092b6d59-d454-41e9-9f87-703efabe4cd0\" (UID: \"092b6d59-d454-41e9-9f87-703efabe4cd0\") "
Sep 18 20:29:28.842488 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:28.842411    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b6d59-d454-41e9-9f87-703efabe4cd0-model-cache\") pod \"092b6d59-d454-41e9-9f87-703efabe4cd0\" (UID: \"092b6d59-d454-41e9-9f87-703efabe4cd0\") "
Sep 18 20:29:28.844306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:28.844272    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092b6d59-d454-41e9-9f87-703efabe4cd0-kube-api-access-clrkm" (OuterVolumeSpecName: "kube-api-access-clrkm") pod "092b6d59-d454-41e9-9f87-703efabe4cd0" (UID: "092b6d59-d454-41e9-9f87-703efabe4cd0"). InnerVolumeSpecName "kube-api-access-clrkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:29:28.864528 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:28.864503    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092b6d59-d454-41e9-9f87-703efabe4cd0-model-cache" (OuterVolumeSpecName: "model-cache") pod "092b6d59-d454-41e9-9f87-703efabe4cd0" (UID: "092b6d59-d454-41e9-9f87-703efabe4cd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:28.943802 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:28.943738    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b6d59-d454-41e9-9f87-703efabe4cd0-model-cache\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:29:28.943802 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:28.943761    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clrkm\" (UniqueName: \"kubernetes.io/projected/092b6d59-d454-41e9-9f87-703efabe4cd0-kube-api-access-clrkm\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:29:29.625847 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:29.625818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5dcw2" event={"ID":"092b6d59-d454-41e9-9f87-703efabe4cd0","Type":"ContainerDied","Data":"d33f0c3d4106d143f0dde02ca2a9ed588f733784b4c80c38ca0f1bde2f4bc226"}
Sep 18 20:29:29.625847 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:29.625846    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5dcw2"
Sep 18 20:29:29.626056 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:29.625852    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d33f0c3d4106d143f0dde02ca2a9ed588f733784b4c80c38ca0f1bde2f4bc226"
Sep 18 20:29:32.247462 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.247431    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-4gd6t"]
Sep 18 20:29:32.247818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.247683    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092b6d59-d454-41e9-9f87-703efabe4cd0" containerName="download-hf-model"
Sep 18 20:29:32.247818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.247694    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="092b6d59-d454-41e9-9f87-703efabe4cd0" containerName="download-hf-model"
Sep 18 20:29:32.247818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.247712    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092b6d59-d454-41e9-9f87-703efabe4cd0" containerName="s3-tls-init-custom"
Sep 18 20:29:32.247818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.247718    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="092b6d59-d454-41e9-9f87-703efabe4cd0" containerName="s3-tls-init-custom"
Sep 18 20:29:32.247818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.247770    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="092b6d59-d454-41e9-9f87-703efabe4cd0" containerName="s3-tls-init-custom"
Sep 18 20:29:32.253295 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.253277    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:32.255733 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.255707    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 20:29:32.255733 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.255728    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 18 20:29:32.258848 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.258822    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4gd6t"]
Sep 18 20:29:32.373856 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.373832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51312b7-3d96-4049-8a47-dd43fa91df57-model-cache\") pod \"s3-tls-init-serving-4gd6t\" (UID: \"e51312b7-3d96-4049-8a47-dd43fa91df57\") " pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:32.373996 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.373867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdngr\" (UniqueName: \"kubernetes.io/projected/e51312b7-3d96-4049-8a47-dd43fa91df57-kube-api-access-jdngr\") pod \"s3-tls-init-serving-4gd6t\" (UID: \"e51312b7-3d96-4049-8a47-dd43fa91df57\") " pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:32.474836 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.474810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51312b7-3d96-4049-8a47-dd43fa91df57-model-cache\") pod \"s3-tls-init-serving-4gd6t\" (UID: \"e51312b7-3d96-4049-8a47-dd43fa91df57\") " pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:32.475000 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.474849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdngr\" (UniqueName: \"kubernetes.io/projected/e51312b7-3d96-4049-8a47-dd43fa91df57-kube-api-access-jdngr\") pod \"s3-tls-init-serving-4gd6t\" (UID: \"e51312b7-3d96-4049-8a47-dd43fa91df57\") " pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:32.475228 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.475207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51312b7-3d96-4049-8a47-dd43fa91df57-model-cache\") pod \"s3-tls-init-serving-4gd6t\" (UID: \"e51312b7-3d96-4049-8a47-dd43fa91df57\") " pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:32.484331 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.484303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdngr\" (UniqueName: \"kubernetes.io/projected/e51312b7-3d96-4049-8a47-dd43fa91df57-kube-api-access-jdngr\") pod \"s3-tls-init-serving-4gd6t\" (UID: \"e51312b7-3d96-4049-8a47-dd43fa91df57\") " pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:32.569846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.569818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:32.842483 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:32.842460    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4gd6t"]
Sep 18 20:29:32.845136 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:29:32.845108    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode51312b7_3d96_4049_8a47_dd43fa91df57.slice/crio-6ef3c067007618fb7e7bf4d2c0c6d9541ba48084294861db57971154eeedd879 WatchSource:0}: Error finding container 6ef3c067007618fb7e7bf4d2c0c6d9541ba48084294861db57971154eeedd879: Status 404 returned error can't find the container with id 6ef3c067007618fb7e7bf4d2c0c6d9541ba48084294861db57971154eeedd879
Sep 18 20:29:33.645049 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:33.645000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gd6t" event={"ID":"e51312b7-3d96-4049-8a47-dd43fa91df57","Type":"ContainerStarted","Data":"5a4e09362373d83851472f5b1e7c148400ec0d85254aec680aaef43160df8d02"}
Sep 18 20:29:33.645049 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:33.645052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gd6t" event={"ID":"e51312b7-3d96-4049-8a47-dd43fa91df57","Type":"ContainerStarted","Data":"6ef3c067007618fb7e7bf4d2c0c6d9541ba48084294861db57971154eeedd879"}
Sep 18 20:29:35.654167 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:35.654134    2566 generic.go:356] "Generic (PLEG): container finished" podID="e51312b7-3d96-4049-8a47-dd43fa91df57" containerID="5a4e09362373d83851472f5b1e7c148400ec0d85254aec680aaef43160df8d02" exitCode=0
Sep 18 20:29:35.654546 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:35.654176    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gd6t" event={"ID":"e51312b7-3d96-4049-8a47-dd43fa91df57","Type":"ContainerDied","Data":"5a4e09362373d83851472f5b1e7c148400ec0d85254aec680aaef43160df8d02"}
Sep 18 20:29:36.662934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:36.662902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gd6t" event={"ID":"e51312b7-3d96-4049-8a47-dd43fa91df57","Type":"ContainerStarted","Data":"64de8e38cd6e92b01f1d37fd3c574c2edabed6cfa3ee54ccaa626e6a255e57e0"}
Sep 18 20:29:36.679150 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:36.679107    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-4gd6t" podStartSLOduration=2.505860787 podStartE2EDuration="4.679092631s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.173231844s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:36.678819278 +0000 UTC m=+545.709745428" watchObservedRunningTime="2026-09-18 20:29:36.679092631 +0000 UTC m=+545.710018771"
Sep 18 20:29:41.682590 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:41.682511    2566 generic.go:356] "Generic (PLEG): container finished" podID="e51312b7-3d96-4049-8a47-dd43fa91df57" containerID="64de8e38cd6e92b01f1d37fd3c574c2edabed6cfa3ee54ccaa626e6a255e57e0" exitCode=0
Sep 18 20:29:41.682947 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:41.682584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gd6t" event={"ID":"e51312b7-3d96-4049-8a47-dd43fa91df57","Type":"ContainerDied","Data":"64de8e38cd6e92b01f1d37fd3c574c2edabed6cfa3ee54ccaa626e6a255e57e0"}
Sep 18 20:29:42.818200 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:42.818172    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:42.964789 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:42.964721    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdngr\" (UniqueName: \"kubernetes.io/projected/e51312b7-3d96-4049-8a47-dd43fa91df57-kube-api-access-jdngr\") pod \"e51312b7-3d96-4049-8a47-dd43fa91df57\" (UID: \"e51312b7-3d96-4049-8a47-dd43fa91df57\") "
Sep 18 20:29:42.964789 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:42.964771    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51312b7-3d96-4049-8a47-dd43fa91df57-model-cache\") pod \"e51312b7-3d96-4049-8a47-dd43fa91df57\" (UID: \"e51312b7-3d96-4049-8a47-dd43fa91df57\") "
Sep 18 20:29:42.966768 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:42.966741    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51312b7-3d96-4049-8a47-dd43fa91df57-kube-api-access-jdngr" (OuterVolumeSpecName: "kube-api-access-jdngr") pod "e51312b7-3d96-4049-8a47-dd43fa91df57" (UID: "e51312b7-3d96-4049-8a47-dd43fa91df57"). InnerVolumeSpecName "kube-api-access-jdngr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:29:42.985956 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:42.985925    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51312b7-3d96-4049-8a47-dd43fa91df57-model-cache" (OuterVolumeSpecName: "model-cache") pod "e51312b7-3d96-4049-8a47-dd43fa91df57" (UID: "e51312b7-3d96-4049-8a47-dd43fa91df57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:43.065770 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:43.065748    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdngr\" (UniqueName: \"kubernetes.io/projected/e51312b7-3d96-4049-8a47-dd43fa91df57-kube-api-access-jdngr\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:29:43.065770 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:43.065770    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51312b7-3d96-4049-8a47-dd43fa91df57-model-cache\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:29:43.690890 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:43.690859    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4gd6t"
Sep 18 20:29:43.691102 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:43.690854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4gd6t" event={"ID":"e51312b7-3d96-4049-8a47-dd43fa91df57","Type":"ContainerDied","Data":"6ef3c067007618fb7e7bf4d2c0c6d9541ba48084294861db57971154eeedd879"}
Sep 18 20:29:43.691102 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:43.690971    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ef3c067007618fb7e7bf4d2c0c6d9541ba48084294861db57971154eeedd879"
Sep 18 20:29:56.303198 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.303161    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"]
Sep 18 20:29:56.303680 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.303460    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e51312b7-3d96-4049-8a47-dd43fa91df57" containerName="s3-tls-init-serving"
Sep 18 20:29:56.303680 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.303472    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e51312b7-3d96-4049-8a47-dd43fa91df57" containerName="s3-tls-init-serving"
Sep 18 20:29:56.303680 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.303484    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e51312b7-3d96-4049-8a47-dd43fa91df57" containerName="download-hf-model"
Sep 18 20:29:56.303680 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.303492    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e51312b7-3d96-4049-8a47-dd43fa91df57" containerName="download-hf-model"
Sep 18 20:29:56.303680 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.303541    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e51312b7-3d96-4049-8a47-dd43fa91df57" containerName="s3-tls-init-serving"
Sep 18 20:29:56.306324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.306309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.308647 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.308624    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:29:56.308776 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.308671    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:29:56.309045 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.309023    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 18 20:29:56.309160 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.309030    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 18 20:29:56.309271 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.309248    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:29:56.309358 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.309253    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:29:56.309769 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.309753    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:29:56.320249 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.320226    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"]
Sep 18 20:29:56.416634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.416601    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"]
Sep 18 20:29:56.419940 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.419922    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.422664 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.422644    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-5c9b8b\""
Sep 18 20:29:56.422954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.422936    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-5c9b8b-predictor-serving-cert\""
Sep 18 20:29:56.423072 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.422987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-5c9b8b-dockercfg-9km4n\""
Sep 18 20:29:56.423127 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.423087    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-5c9b8b-kube-rbac-proxy-sar-config\""
Sep 18 20:29:56.431415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.431394    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"]
Sep 18 20:29:56.463625 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.463602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e68a93-2192-4dd3-ba52-697e1c2f0657-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.463713 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.463633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.463713 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.463665    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.463784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.463773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvlj\" (UniqueName: \"kubernetes.io/projected/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-api-access-6lvlj\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.463818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.463803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e68a93-2192-4dd3-ba52-697e1c2f0657-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.564833 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.564768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvlj\" (UniqueName: \"kubernetes.io/projected/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-api-access-6lvlj\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.564833 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.564806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-proxy-tls\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.564833 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.564830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e68a93-2192-4dd3-ba52-697e1c2f0657-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.565059 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.564856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e68a93-2192-4dd3-ba52-697e1c2f0657-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.565059 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.564876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92qp\" (UniqueName: \"kubernetes.io/projected/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-api-access-t92qp\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.565059 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.564894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.565059 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.564936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.565059 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.564963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.565059 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.565008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kserve-provision-location\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.565362 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.565076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-cabundle-cert\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.565362 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.565221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e68a93-2192-4dd3-ba52-697e1c2f0657-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.565571 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.565554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.565612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.565574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.567294 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.567274    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e68a93-2192-4dd3-ba52-697e1c2f0657-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.572853 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.572831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvlj\" (UniqueName: \"kubernetes.io/projected/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-api-access-6lvlj\") pod \"isvc-sklearn-batcher-predictor-78888c66b-4xjr7\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.616759 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.616739    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:29:56.667378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.666601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-cabundle-cert\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.667378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.666664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-proxy-tls\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.667378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.666710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92qp\" (UniqueName: \"kubernetes.io/projected/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-api-access-t92qp\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.667378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.666751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.667378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.666796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kserve-provision-location\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.667378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.667211    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kserve-provision-location\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.667838 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.667810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-cabundle-cert\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.668378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.668353    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.671955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.670633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-proxy-tls\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.678733 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.678670    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92qp\" (UniqueName: \"kubernetes.io/projected/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-api-access-t92qp\") pod \"isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") " pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.730502 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.730468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:29:56.889624 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:56.889599    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"]
Sep 18 20:29:56.892326 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:29:56.892303    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e68a93_2192_4dd3_ba52_697e1c2f0657.slice/crio-6a3c84f45d8303949dce217d9c1f317e1943c9c1b9eac29769d535e3e3ef9018 WatchSource:0}: Error finding container 6a3c84f45d8303949dce217d9c1f317e1943c9c1b9eac29769d535e3e3ef9018: Status 404 returned error can't find the container with id 6a3c84f45d8303949dce217d9c1f317e1943c9c1b9eac29769d535e3e3ef9018
Sep 18 20:29:57.002699 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:57.002674    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"]
Sep 18 20:29:57.004447 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:29:57.004424    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845dbf03_6157_4e37_b1ee_8f2ecb705ffc.slice/crio-8b579bd0fc57896a6868ddd67de7a6f639ec28172d1ad9476e35f5e1489b494d WatchSource:0}: Error finding container 8b579bd0fc57896a6868ddd67de7a6f639ec28172d1ad9476e35f5e1489b494d: Status 404 returned error can't find the container with id 8b579bd0fc57896a6868ddd67de7a6f639ec28172d1ad9476e35f5e1489b494d
Sep 18 20:29:57.744873 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:57.744832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9" event={"ID":"845dbf03-6157-4e37-b1ee-8f2ecb705ffc","Type":"ContainerStarted","Data":"6c99d8aca3dae12da6718586de17da17bcf94371d0de3845e4efc8db73935db1"}
Sep 18 20:29:57.745347 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:57.744942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9" event={"ID":"845dbf03-6157-4e37-b1ee-8f2ecb705ffc","Type":"ContainerStarted","Data":"8b579bd0fc57896a6868ddd67de7a6f639ec28172d1ad9476e35f5e1489b494d"}
Sep 18 20:29:57.746536 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:57.746508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerStarted","Data":"da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0"}
Sep 18 20:29:57.746643 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:29:57.746584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerStarted","Data":"6a3c84f45d8303949dce217d9c1f317e1943c9c1b9eac29769d535e3e3ef9018"}
Sep 18 20:30:01.763566 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:01.763490    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9_845dbf03-6157-4e37-b1ee-8f2ecb705ffc/storage-initializer/0.log"
Sep 18 20:30:01.763566 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:01.763526    2566 generic.go:356] "Generic (PLEG): container finished" podID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" containerID="6c99d8aca3dae12da6718586de17da17bcf94371d0de3845e4efc8db73935db1" exitCode=1
Sep 18 20:30:01.763967 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:01.763585    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9" event={"ID":"845dbf03-6157-4e37-b1ee-8f2ecb705ffc","Type":"ContainerDied","Data":"6c99d8aca3dae12da6718586de17da17bcf94371d0de3845e4efc8db73935db1"}
Sep 18 20:30:01.765348 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:01.765323    2566 generic.go:356] "Generic (PLEG): container finished" podID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerID="da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0" exitCode=0
Sep 18 20:30:01.765417 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:01.765394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerDied","Data":"da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0"}
Sep 18 20:30:02.777992 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:02.777898    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9_845dbf03-6157-4e37-b1ee-8f2ecb705ffc/storage-initializer/0.log"
Sep 18 20:30:02.778548 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:02.777999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9" event={"ID":"845dbf03-6157-4e37-b1ee-8f2ecb705ffc","Type":"ContainerStarted","Data":"c0d6a09b66d925b808387d533598b108eb044ac0d567c1ec6e2cca49e435fdbf"}
Sep 18 20:30:06.108112 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:06.107535    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"]
Sep 18 20:30:06.108112 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:06.107781    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9" podUID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" containerName="storage-initializer" containerID="cri-o://c0d6a09b66d925b808387d533598b108eb044ac0d567c1ec6e2cca49e435fdbf" gracePeriod=30
Sep 18 20:30:06.803888 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:06.803846    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9_845dbf03-6157-4e37-b1ee-8f2ecb705ffc/storage-initializer/1.log"
Sep 18 20:30:06.804389 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:06.804368    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9_845dbf03-6157-4e37-b1ee-8f2ecb705ffc/storage-initializer/0.log"
Sep 18 20:30:06.804507 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:06.804405    2566 generic.go:356] "Generic (PLEG): container finished" podID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" containerID="c0d6a09b66d925b808387d533598b108eb044ac0d567c1ec6e2cca49e435fdbf" exitCode=1
Sep 18 20:30:06.804507 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:06.804467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9" event={"ID":"845dbf03-6157-4e37-b1ee-8f2ecb705ffc","Type":"ContainerDied","Data":"c0d6a09b66d925b808387d533598b108eb044ac0d567c1ec6e2cca49e435fdbf"}
Sep 18 20:30:06.804622 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:06.804551    2566 scope.go:117] "RemoveContainer" containerID="6c99d8aca3dae12da6718586de17da17bcf94371d0de3845e4efc8db73935db1"
Sep 18 20:30:07.046107 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.046083    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9_845dbf03-6157-4e37-b1ee-8f2ecb705ffc/storage-initializer/1.log"
Sep 18 20:30:07.049590 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.049558    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:30:07.162196 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.162164    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-proxy-tls\") pod \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") "
Sep 18 20:30:07.162648 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.162230    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kserve-provision-location\") pod \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") "
Sep 18 20:30:07.162648 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.162304    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t92qp\" (UniqueName: \"kubernetes.io/projected/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-api-access-t92qp\") pod \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") "
Sep 18 20:30:07.162648 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.162390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-rbac-proxy-sar-config\") pod \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") "
Sep 18 20:30:07.162648 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.162442    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-cabundle-cert\") pod \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\" (UID: \"845dbf03-6157-4e37-b1ee-8f2ecb705ffc\") "
Sep 18 20:30:07.162648 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.162492    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "845dbf03-6157-4e37-b1ee-8f2ecb705ffc" (UID: "845dbf03-6157-4e37-b1ee-8f2ecb705ffc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:07.162901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.162725    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:30:07.162901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.162752    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "845dbf03-6157-4e37-b1ee-8f2ecb705ffc" (UID: "845dbf03-6157-4e37-b1ee-8f2ecb705ffc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:07.163000 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.162916    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "845dbf03-6157-4e37-b1ee-8f2ecb705ffc" (UID: "845dbf03-6157-4e37-b1ee-8f2ecb705ffc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:07.164795 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.164769    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "845dbf03-6157-4e37-b1ee-8f2ecb705ffc" (UID: "845dbf03-6157-4e37-b1ee-8f2ecb705ffc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:30:07.164902 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.164811    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-api-access-t92qp" (OuterVolumeSpecName: "kube-api-access-t92qp") pod "845dbf03-6157-4e37-b1ee-8f2ecb705ffc" (UID: "845dbf03-6157-4e37-b1ee-8f2ecb705ffc"). InnerVolumeSpecName "kube-api-access-t92qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:07.263697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.263659    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t92qp\" (UniqueName: \"kubernetes.io/projected/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-api-access-t92qp\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:30:07.263697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.263697    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:30:07.263900 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.263716    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:30:07.263900 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.263730    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845dbf03-6157-4e37-b1ee-8f2ecb705ffc-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:30:07.810810 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.810778    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9_845dbf03-6157-4e37-b1ee-8f2ecb705ffc/storage-initializer/1.log"
Sep 18 20:30:07.810992 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.810844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9" event={"ID":"845dbf03-6157-4e37-b1ee-8f2ecb705ffc","Type":"ContainerDied","Data":"8b579bd0fc57896a6868ddd67de7a6f639ec28172d1ad9476e35f5e1489b494d"}
Sep 18 20:30:07.810992 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.810916    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"
Sep 18 20:30:07.810992 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.810936    2566 scope.go:117] "RemoveContainer" containerID="c0d6a09b66d925b808387d533598b108eb044ac0d567c1ec6e2cca49e435fdbf"
Sep 18 20:30:07.844510 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.844477    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"]
Sep 18 20:30:07.847505 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:07.847482    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5c9b8b-predictor-7c74f49864-spww9"]
Sep 18 20:30:09.460249 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:09.460214    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" path="/var/lib/kubelet/pods/845dbf03-6157-4e37-b1ee-8f2ecb705ffc/volumes"
Sep 18 20:30:15.849979 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:15.849950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerStarted","Data":"fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9"}
Sep 18 20:30:18.864490 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:18.864452    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerStarted","Data":"24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743"}
Sep 18 20:30:20.876631 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:20.876595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerStarted","Data":"d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773"}
Sep 18 20:30:20.877054 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:20.876807    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:30:20.877054 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:20.876865    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:30:20.877054 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:20.876899    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:30:20.878406 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:20.878378    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:30:20.879056 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:20.879035    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:20.899209 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:20.899164    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podStartSLOduration=2.09348306 podStartE2EDuration="24.8991529s" totalImagesPullingTime="18.863660302s" totalInitContainerRuntime="3.942009538s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:30:20.89745452 +0000 UTC m=+589.928380672" watchObservedRunningTime="2026-09-18 20:30:20.8991529 +0000 UTC m=+589.930079040"
Sep 18 20:30:21.881135 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:21.881101    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:30:21.881599 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:21.881575    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:26.886199 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:26.886169    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:30:26.886825 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:26.886792    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:30:26.887162 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:26.887136    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:31.371316 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:31.371280    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:30:31.371810 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:31.371392    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:30:36.887443 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:36.887399    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:30:36.887868 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:36.887843    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:46.886899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:46.886852    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:30:46.887375 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:46.887286    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:30:56.887660 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:56.887566    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:30:56.888083 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:30:56.888058    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:06.886931 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:06.886891    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:31:06.887400 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:06.887332    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:16.887620 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:16.887578    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:31:16.890020 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:16.888046    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:26.887745 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:26.887695    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:31:26.888139 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:26.888086    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:33.459190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:33.459161    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:31:33.459559 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:33.459248    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:31:41.668504 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.668465    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"]
Sep 18 20:31:41.668995 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.668728    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" containerID="cri-o://fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9" gracePeriod=30
Sep 18 20:31:41.668995 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.668763    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy" containerID="cri-o://24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743" gracePeriod=30
Sep 18 20:31:41.668995 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.668756    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" containerID="cri-o://d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773" gracePeriod=30
Sep 18 20:31:41.750076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.750038    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"]
Sep 18 20:31:41.750378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.750363    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" containerName="storage-initializer"
Sep 18 20:31:41.750460 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.750382    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" containerName="storage-initializer"
Sep 18 20:31:41.750520 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.750482    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" containerName="storage-initializer"
Sep 18 20:31:41.750520 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.750496    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" containerName="storage-initializer"
Sep 18 20:31:41.750627 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.750579    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" containerName="storage-initializer"
Sep 18 20:31:41.750627 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.750589    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="845dbf03-6157-4e37-b1ee-8f2ecb705ffc" containerName="storage-initializer"
Sep 18 20:31:41.753727 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.753709    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.756338 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.756298    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 18 20:31:41.756338 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.756302    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 18 20:31:41.763748 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.763724    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"]
Sep 18 20:31:41.819639 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.819612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwbq4\" (UniqueName: \"kubernetes.io/projected/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-api-access-vwbq4\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.819760 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.819662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.819760 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.819721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.819845 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.819763    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.819845 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.819800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.881933 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.881900    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 18 20:31:41.920222 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.920162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.920222 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.920197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.920413 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.920238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.920413 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.920275    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwbq4\" (UniqueName: \"kubernetes.io/projected/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-api-access-vwbq4\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.920413 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.920327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.920756 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.920736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.920926 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.920888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.921061 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.920905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.922662 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.922643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:41.929280 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:41.929261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwbq4\" (UniqueName: \"kubernetes.io/projected/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-api-access-vwbq4\") pod \"isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:42.064477 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:42.064442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:42.202453 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:42.202374    2566 generic.go:356] "Generic (PLEG): container finished" podID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerID="24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743" exitCode=2
Sep 18 20:31:42.202453 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:42.202446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerDied","Data":"24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743"}
Sep 18 20:31:42.337442 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:42.337383    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"]
Sep 18 20:31:42.340042 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:31:42.340003    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9d7e0a_7a45_416e_a5e2_e06b3e3d8db3.slice/crio-8a86483cbd9804fa978e576031a47d27e571200d9c05d890543d5a50016e4ed4 WatchSource:0}: Error finding container 8a86483cbd9804fa978e576031a47d27e571200d9c05d890543d5a50016e4ed4: Status 404 returned error can't find the container with id 8a86483cbd9804fa978e576031a47d27e571200d9c05d890543d5a50016e4ed4
Sep 18 20:31:43.210158 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:43.210122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerStarted","Data":"153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b"}
Sep 18 20:31:43.210512 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:43.210183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerStarted","Data":"8a86483cbd9804fa978e576031a47d27e571200d9c05d890543d5a50016e4ed4"}
Sep 18 20:31:43.455236 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:43.455190    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:31:43.455587 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:43.455560    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:46.224691 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:46.224665    2566 generic.go:356] "Generic (PLEG): container finished" podID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerID="fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9" exitCode=0
Sep 18 20:31:46.224983 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:46.224700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerDied","Data":"fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9"}
Sep 18 20:31:46.881739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:46.881698    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 18 20:31:47.230151 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:47.230066    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerID="153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b" exitCode=0
Sep 18 20:31:47.230502 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:47.230145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerDied","Data":"153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b"}
Sep 18 20:31:48.236323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:48.236287    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerStarted","Data":"dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1"}
Sep 18 20:31:48.236708 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:48.236357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerStarted","Data":"90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313"}
Sep 18 20:31:48.236708 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:48.236383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerStarted","Data":"aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a"}
Sep 18 20:31:48.236708 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:48.236671    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:48.236845 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:48.236813    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:48.238111 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:48.238084    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:31:48.259331 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:48.259286    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podStartSLOduration=3.308027494 podStartE2EDuration="7.259272666s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.951245172s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:31:48.256697749 +0000 UTC m=+677.287623890" watchObservedRunningTime="2026-09-18 20:31:48.259272666 +0000 UTC m=+677.290198806"
Sep 18 20:31:49.240689 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:49.240647    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:31:49.241642 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:49.241611    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:49.245717 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:49.245676    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:50.244518 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:50.244477    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:31:50.244908 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:50.244843    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:51.881281 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:51.881242    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 18 20:31:51.881688 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:51.881394    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:31:53.455954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:53.455910    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:31:53.456443 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:53.456257    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:55.249074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:55.249046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:31:55.249619 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:55.249590    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:31:55.249972 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:55.249948    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:31:56.882080 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:31:56.882044    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 18 20:32:01.881350 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:01.881310    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 18 20:32:03.456030 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:03.455971    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 20:32:03.456415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:03.456325    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:32:03.459454 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:03.459426    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:32:03.459592 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:03.459545    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:32:05.250331 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:05.250278    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:32:05.250755 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:05.250724    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:32:06.881428 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:06.881389    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 18 20:32:11.813694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.813671    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:32:11.869524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.869496    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lvlj\" (UniqueName: \"kubernetes.io/projected/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-api-access-6lvlj\") pod \"10e68a93-2192-4dd3-ba52-697e1c2f0657\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") "
Sep 18 20:32:11.869725 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.869532    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e68a93-2192-4dd3-ba52-697e1c2f0657-proxy-tls\") pod \"10e68a93-2192-4dd3-ba52-697e1c2f0657\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") "
Sep 18 20:32:11.869725 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.869574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e68a93-2192-4dd3-ba52-697e1c2f0657-kserve-provision-location\") pod \"10e68a93-2192-4dd3-ba52-697e1c2f0657\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") "
Sep 18 20:32:11.869725 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.869600    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-cabundle-cert\") pod \"10e68a93-2192-4dd3-ba52-697e1c2f0657\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") "
Sep 18 20:32:11.869725 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.869626    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-rbac-proxy-sar-config\") pod \"10e68a93-2192-4dd3-ba52-697e1c2f0657\" (UID: \"10e68a93-2192-4dd3-ba52-697e1c2f0657\") "
Sep 18 20:32:11.869962 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.869928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e68a93-2192-4dd3-ba52-697e1c2f0657-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10e68a93-2192-4dd3-ba52-697e1c2f0657" (UID: "10e68a93-2192-4dd3-ba52-697e1c2f0657"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:11.870001 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.869953    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "10e68a93-2192-4dd3-ba52-697e1c2f0657" (UID: "10e68a93-2192-4dd3-ba52-697e1c2f0657"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:11.870100 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.870074    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "10e68a93-2192-4dd3-ba52-697e1c2f0657" (UID: "10e68a93-2192-4dd3-ba52-697e1c2f0657"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:11.871669 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.871646    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-api-access-6lvlj" (OuterVolumeSpecName: "kube-api-access-6lvlj") pod "10e68a93-2192-4dd3-ba52-697e1c2f0657" (UID: "10e68a93-2192-4dd3-ba52-697e1c2f0657"). InnerVolumeSpecName "kube-api-access-6lvlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:11.871764 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.871739    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e68a93-2192-4dd3-ba52-697e1c2f0657-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10e68a93-2192-4dd3-ba52-697e1c2f0657" (UID: "10e68a93-2192-4dd3-ba52-697e1c2f0657"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:32:11.970580 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.970513    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lvlj\" (UniqueName: \"kubernetes.io/projected/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-api-access-6lvlj\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:32:11.970580 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.970537    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e68a93-2192-4dd3-ba52-697e1c2f0657-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:32:11.970580 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.970551    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e68a93-2192-4dd3-ba52-697e1c2f0657-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:32:11.970580 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.970562    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:32:11.970580 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:11.970574    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e68a93-2192-4dd3-ba52-697e1c2f0657-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:32:12.333916 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.333881    2566 generic.go:356] "Generic (PLEG): container finished" podID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerID="d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773" exitCode=0
Sep 18 20:32:12.334116 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.333921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerDied","Data":"d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773"}
Sep 18 20:32:12.334116 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.333947    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"
Sep 18 20:32:12.334116 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.333988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7" event={"ID":"10e68a93-2192-4dd3-ba52-697e1c2f0657","Type":"ContainerDied","Data":"6a3c84f45d8303949dce217d9c1f317e1943c9c1b9eac29769d535e3e3ef9018"}
Sep 18 20:32:12.334116 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.334047    2566 scope.go:117] "RemoveContainer" containerID="d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773"
Sep 18 20:32:12.342551 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.342530    2566 scope.go:117] "RemoveContainer" containerID="24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743"
Sep 18 20:32:12.349837 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.349815    2566 scope.go:117] "RemoveContainer" containerID="fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9"
Sep 18 20:32:12.356750 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.356684    2566 scope.go:117] "RemoveContainer" containerID="da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0"
Sep 18 20:32:12.358502 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.358478    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"]
Sep 18 20:32:12.361978 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.361958    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-78888c66b-4xjr7"]
Sep 18 20:32:12.364432 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.364418    2566 scope.go:117] "RemoveContainer" containerID="d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773"
Sep 18 20:32:12.364702 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:32:12.364685    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773\": container with ID starting with d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773 not found: ID does not exist" containerID="d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773"
Sep 18 20:32:12.364770 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.364708    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773"} err="failed to get container status \"d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773\": rpc error: code = NotFound desc = could not find container \"d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773\": container with ID starting with d3de84deb708c68552bd6ca9d4292f77cfc9cfd11a2d0f0a4ddb10516c96c773 not found: ID does not exist"
Sep 18 20:32:12.364770 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.364724    2566 scope.go:117] "RemoveContainer" containerID="24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743"
Sep 18 20:32:12.364969 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:32:12.364953    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743\": container with ID starting with 24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743 not found: ID does not exist" containerID="24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743"
Sep 18 20:32:12.365008 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.364973    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743"} err="failed to get container status \"24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743\": rpc error: code = NotFound desc = could not find container \"24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743\": container with ID starting with 24df2eba8ba9d8769cabdff79d9dcd46f9908996d0134759dae784025d678743 not found: ID does not exist"
Sep 18 20:32:12.365008 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.364986    2566 scope.go:117] "RemoveContainer" containerID="fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9"
Sep 18 20:32:12.365242 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:32:12.365228    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9\": container with ID starting with fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9 not found: ID does not exist" containerID="fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9"
Sep 18 20:32:12.365292 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.365245    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9"} err="failed to get container status \"fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9\": rpc error: code = NotFound desc = could not find container \"fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9\": container with ID starting with fe832468a375e6885454b2be39bb7a7547a570ff73b2a9fe73281f2f32c2d4d9 not found: ID does not exist"
Sep 18 20:32:12.365292 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.365258    2566 scope.go:117] "RemoveContainer" containerID="da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0"
Sep 18 20:32:12.365500 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:32:12.365482    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0\": container with ID starting with da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0 not found: ID does not exist" containerID="da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0"
Sep 18 20:32:12.365540 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:12.365506    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0"} err="failed to get container status \"da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0\": rpc error: code = NotFound desc = could not find container \"da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0\": container with ID starting with da1c8ce8db614b6494e77efb36acde7de16fa57d78deae5c11bf2f86b082e1c0 not found: ID does not exist"
Sep 18 20:32:13.459658 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:13.459627    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" path="/var/lib/kubelet/pods/10e68a93-2192-4dd3-ba52-697e1c2f0657/volumes"
Sep 18 20:32:15.250267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:15.250222    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:32:15.250724 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:15.250705    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:32:25.250403 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:25.250317    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:32:25.250803 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:25.250783    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:32:35.250422 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:35.250380    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:32:35.250850 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:35.250820    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:32:45.249750 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:45.249711    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:32:45.250187 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:45.250148    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:32:55.250426 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:55.250382    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:32:55.250836 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:32:55.250768    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:05.250188 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:05.250159    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:33:05.250718 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:05.250246    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:33:17.292217 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.292183    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"]
Sep 18 20:33:17.292746 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.292441    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" containerID="cri-o://aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a" gracePeriod=30
Sep 18 20:33:17.292746 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.292456    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" containerID="cri-o://dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1" gracePeriod=30
Sep 18 20:33:17.292746 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.292565    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy" containerID="cri-o://90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313" gracePeriod=30
Sep 18 20:33:17.406392 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406353    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"]
Sep 18 20:33:17.406676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406663    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="storage-initializer"
Sep 18 20:33:17.406676 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406677    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="storage-initializer"
Sep 18 20:33:17.406760 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406687    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container"
Sep 18 20:33:17.406760 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406693    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container"
Sep 18 20:33:17.406760 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406701    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy"
Sep 18 20:33:17.406760 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406707    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy"
Sep 18 20:33:17.406760 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406716    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent"
Sep 18 20:33:17.406760 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406724    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent"
Sep 18 20:33:17.406942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406781    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="agent"
Sep 18 20:33:17.406942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406789    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kube-rbac-proxy"
Sep 18 20:33:17.406942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.406798    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e68a93-2192-4dd3-ba52-697e1c2f0657" containerName="kserve-container"
Sep 18 20:33:17.409787 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.409768    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.412296 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.412279    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 18 20:33:17.412481 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.412468    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 18 20:33:17.422304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.422279    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"]
Sep 18 20:33:17.475108 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.475076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-68mb5\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.475255 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.475193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-68mb5\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.475332 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.475267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswpt\" (UniqueName: \"kubernetes.io/projected/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-api-access-jswpt\") pod \"message-dumper-predictor-6575bcfdfd-68mb5\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.576193 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.576113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-68mb5\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.576193 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.576153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-68mb5\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.576403 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.576201    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswpt\" (UniqueName: \"kubernetes.io/projected/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-api-access-jswpt\") pod \"message-dumper-predictor-6575bcfdfd-68mb5\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.576720 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.576696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-68mb5\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.578488 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.578465    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-68mb5\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.586441 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.586413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswpt\" (UniqueName: \"kubernetes.io/projected/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-api-access-jswpt\") pod \"message-dumper-predictor-6575bcfdfd-68mb5\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.590088 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.590065    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerID="90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313" exitCode=2
Sep 18 20:33:17.590222 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.590132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerDied","Data":"90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313"}
Sep 18 20:33:17.724814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.724789    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:17.997366 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:17.997340    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"]
Sep 18 20:33:17.999750 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:33:17.999724    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff437fe3_a453_4c05_a70b_d3a4dcd31faf.slice/crio-49d00c8d1532f9919d74f28f3cd35497542979eb39e4c2bb04a9563937c7c297 WatchSource:0}: Error finding container 49d00c8d1532f9919d74f28f3cd35497542979eb39e4c2bb04a9563937c7c297: Status 404 returned error can't find the container with id 49d00c8d1532f9919d74f28f3cd35497542979eb39e4c2bb04a9563937c7c297
Sep 18 20:33:18.001588 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:18.001574    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:33:18.595644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:18.595612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5" event={"ID":"ff437fe3-a453-4c05-a70b-d3a4dcd31faf","Type":"ContainerStarted","Data":"49d00c8d1532f9919d74f28f3cd35497542979eb39e4c2bb04a9563937c7c297"}
Sep 18 20:33:19.601113 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:19.601076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5" event={"ID":"ff437fe3-a453-4c05-a70b-d3a4dcd31faf","Type":"ContainerStarted","Data":"3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0"}
Sep 18 20:33:19.601512 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:19.601137    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:19.601512 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:19.601165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5" event={"ID":"ff437fe3-a453-4c05-a70b-d3a4dcd31faf","Type":"ContainerStarted","Data":"d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f"}
Sep 18 20:33:19.601512 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:19.601186    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:19.603090 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:19.603069    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:19.620610 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:19.620532    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5" podStartSLOduration=1.252543445 podStartE2EDuration="2.620518715s" totalImagesPullingTime="1.36797527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:33:19.619336645 +0000 UTC m=+768.650262788" watchObservedRunningTime="2026-09-18 20:33:19.620518715 +0000 UTC m=+768.651444855"
Sep 18 20:33:20.244680 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:20.244643    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 18 20:33:22.617434 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:22.617400    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerID="aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a" exitCode=0
Sep 18 20:33:22.617798 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:22.617457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerDied","Data":"aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a"}
Sep 18 20:33:25.245531 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:25.245491    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 18 20:33:25.249861 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:25.249835    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:33:25.250297 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:25.250273    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:26.616521 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:26.616493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:33:27.446299 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.446269    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"]
Sep 18 20:33:27.450050 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.450030    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.452743 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.452718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 18 20:33:27.452834 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.452726    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 18 20:33:27.463671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.463647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"]
Sep 18 20:33:27.558185 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.558151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d35638c-5ab8-4d2c-a9bc-99da6989727c-proxy-tls\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.558336 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.558216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kserve-provision-location\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.558336 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.558248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-cabundle-cert\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.558336 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.558289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctxc\" (UniqueName: \"kubernetes.io/projected/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-api-access-cctxc\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.558467 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.558349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.659251 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.659215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d35638c-5ab8-4d2c-a9bc-99da6989727c-proxy-tls\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.659671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.659304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kserve-provision-location\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.659671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.659341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-cabundle-cert\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.659671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.659394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctxc\" (UniqueName: \"kubernetes.io/projected/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-api-access-cctxc\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.659671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.659440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.660294 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.660267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.663330 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.663304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d35638c-5ab8-4d2c-a9bc-99da6989727c-proxy-tls\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.663639 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.663616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kserve-provision-location\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.664230 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.664202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-cabundle-cert\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.672974 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.672956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctxc\" (UniqueName: \"kubernetes.io/projected/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-api-access-cctxc\") pod \"isvc-logger-predictor-76f676cbf7-clkgz\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:27.761188 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:27.761119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:28.034424 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:28.034401    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"]
Sep 18 20:33:28.036570 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:33:28.036537    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d35638c_5ab8_4d2c_a9bc_99da6989727c.slice/crio-67cbb0dc949190aabbe62811757df1e378b66f20295304b0dd82806d12db7a76 WatchSource:0}: Error finding container 67cbb0dc949190aabbe62811757df1e378b66f20295304b0dd82806d12db7a76: Status 404 returned error can't find the container with id 67cbb0dc949190aabbe62811757df1e378b66f20295304b0dd82806d12db7a76
Sep 18 20:33:28.644371 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:28.644334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerStarted","Data":"17a3ebcb02f1f256c93a365fab1b2629d0404694dcfb0958f88c42c8feb50856"}
Sep 18 20:33:28.644564 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:28.644401    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerStarted","Data":"67cbb0dc949190aabbe62811757df1e378b66f20295304b0dd82806d12db7a76"}
Sep 18 20:33:30.244806 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:30.244767    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 18 20:33:30.245286 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:30.244899    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:33:32.662326 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:32.662293    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerID="17a3ebcb02f1f256c93a365fab1b2629d0404694dcfb0958f88c42c8feb50856" exitCode=0
Sep 18 20:33:32.662690 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:32.662367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerDied","Data":"17a3ebcb02f1f256c93a365fab1b2629d0404694dcfb0958f88c42c8feb50856"}
Sep 18 20:33:33.669246 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:33.669205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerStarted","Data":"c459144f8439d5c16ae97f0b6effff579b03d8dfde89aacaa596c7e6c3f2c99c"}
Sep 18 20:33:33.669616 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:33.669278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerStarted","Data":"008c9c08d20a99adfd6f871512f144b6178e55864b4d09f278a65f4e54046ad0"}
Sep 18 20:33:33.669616 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:33.669305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerStarted","Data":"164ab6a002fea01ee0aa2fb608d092d4cc4adc8e5711e5eb62bf055be517f56b"}
Sep 18 20:33:33.669716 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:33.669636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:33.669836 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:33.669819    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:33.671207 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:33.671181    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:33.690642 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:33.690598    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podStartSLOduration=2.968721942 podStartE2EDuration="6.690586234s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721864292s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:33:33.688485437 +0000 UTC m=+782.719411577" watchObservedRunningTime="2026-09-18 20:33:33.690586234 +0000 UTC m=+782.721512374"
Sep 18 20:33:34.673415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:34.673376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:34.673860 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:34.673568    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:34.674767 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:34.674739    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:35.245211 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:35.245167    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 18 20:33:35.249598 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:35.249567    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:33:35.249711 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:35.249663    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:35.677458 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:35.677414    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:35.677864 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:35.677798    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:40.244874 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:40.244825    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 18 20:33:40.682445 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:40.682417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:33:40.683175 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:40.683135    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:40.683362 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:40.683335    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:45.244804 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:45.244765    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 18 20:33:45.250125 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:45.250083    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Sep 18 20:33:45.250252 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:45.250231    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:33:45.250452 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:45.250425    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:33:45.250530 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:45.250518    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:33:47.434703 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.434681    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:33:47.523386 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.523349    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kserve-provision-location\") pod \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") "
Sep 18 20:33:47.523569 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.523387    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwbq4\" (UniqueName: \"kubernetes.io/projected/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-api-access-vwbq4\") pod \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") "
Sep 18 20:33:47.523569 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.523442    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-rbac-proxy-sar-config\") pod \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") "
Sep 18 20:33:47.523569 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.523513    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-proxy-tls\") pod \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") "
Sep 18 20:33:47.523569 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.523549    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-cabundle-cert\") pod \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\" (UID: \"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3\") "
Sep 18 20:33:47.523809 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.523696    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" (UID: "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:33:47.524009 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.523828    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:33:47.524121 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.524042    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" (UID: "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:47.524296 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.524266    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" (UID: "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:47.527747 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.527718    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-api-access-vwbq4" (OuterVolumeSpecName: "kube-api-access-vwbq4") pod "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" (UID: "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3"). InnerVolumeSpecName "kube-api-access-vwbq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:33:47.530657 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.530635    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" (UID: "fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:33:47.625324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.625253    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:33:47.625324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.625280    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:33:47.625324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.625288    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwbq4\" (UniqueName: \"kubernetes.io/projected/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-api-access-vwbq4\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:33:47.625324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.625297    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:33:47.726156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.726127    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerID="dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1" exitCode=0
Sep 18 20:33:47.726307 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.726184    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"
Sep 18 20:33:47.726307 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.726198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerDied","Data":"dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1"}
Sep 18 20:33:47.726307 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.726264    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh" event={"ID":"fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3","Type":"ContainerDied","Data":"8a86483cbd9804fa978e576031a47d27e571200d9c05d890543d5a50016e4ed4"}
Sep 18 20:33:47.726307 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.726297    2566 scope.go:117] "RemoveContainer" containerID="dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1"
Sep 18 20:33:47.734187 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.734169    2566 scope.go:117] "RemoveContainer" containerID="90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313"
Sep 18 20:33:47.741355 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.741338    2566 scope.go:117] "RemoveContainer" containerID="aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a"
Sep 18 20:33:47.747977 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.747951    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"]
Sep 18 20:33:47.748719 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.748704    2566 scope.go:117] "RemoveContainer" containerID="153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b"
Sep 18 20:33:47.753158 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.753139    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67589f7bc6-s2fjh"]
Sep 18 20:33:47.756880 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.756864    2566 scope.go:117] "RemoveContainer" containerID="dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1"
Sep 18 20:33:47.757137 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:33:47.757122    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1\": container with ID starting with dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1 not found: ID does not exist" containerID="dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1"
Sep 18 20:33:47.757192 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.757145    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1"} err="failed to get container status \"dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1\": rpc error: code = NotFound desc = could not find container \"dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1\": container with ID starting with dc4df5a7e9943579412203eefe327bc0f90b00a935552c8edc6f63b8022936f1 not found: ID does not exist"
Sep 18 20:33:47.757192 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.757161    2566 scope.go:117] "RemoveContainer" containerID="90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313"
Sep 18 20:33:47.757365 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:33:47.757351    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313\": container with ID starting with 90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313 not found: ID does not exist" containerID="90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313"
Sep 18 20:33:47.757419 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.757368    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313"} err="failed to get container status \"90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313\": rpc error: code = NotFound desc = could not find container \"90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313\": container with ID starting with 90432ef69b884b1ea0d7f27fb1108a29c7f77b18b3ee2572e698c6996e9fb313 not found: ID does not exist"
Sep 18 20:33:47.757419 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.757382    2566 scope.go:117] "RemoveContainer" containerID="aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a"
Sep 18 20:33:47.757666 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:33:47.757647    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a\": container with ID starting with aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a not found: ID does not exist" containerID="aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a"
Sep 18 20:33:47.757714 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.757670    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a"} err="failed to get container status \"aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a\": rpc error: code = NotFound desc = could not find container \"aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a\": container with ID starting with aba76b6aa33738f7669909570ed09e36f3b70c02c2e5f39e45970b2dbc66314a not found: ID does not exist"
Sep 18 20:33:47.757714 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.757686    2566 scope.go:117] "RemoveContainer" containerID="153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b"
Sep 18 20:33:47.757908 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:33:47.757888    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b\": container with ID starting with 153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b not found: ID does not exist" containerID="153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b"
Sep 18 20:33:47.757947 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:47.757911    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b"} err="failed to get container status \"153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b\": rpc error: code = NotFound desc = could not find container \"153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b\": container with ID starting with 153bce5e5e757c90728fe9933a6d6d58d142a62fd020dc031ed9ead4de36429b not found: ID does not exist"
Sep 18 20:33:49.459981 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:49.459947    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" path="/var/lib/kubelet/pods/fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3/volumes"
Sep 18 20:33:50.683753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:50.683707    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:33:50.684243 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:33:50.684217    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:34:00.683552 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:00.683462    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:34:00.683930 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:00.683908    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:34:10.683165 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:10.683119    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:34:10.683628 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:10.683602    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:34:20.683357 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:20.683314    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:34:20.683818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:20.683715    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:34:30.683124 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:30.683079    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:34:30.683501 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:30.683430    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:34:40.683258 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:40.683204    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:34:40.683667 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:40.683575    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:34:50.684009 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:50.683980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:34:50.684433 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:34:50.684156    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:35:02.935170 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:02.935135    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-68mb5_ff437fe3-a453-4c05-a70b-d3a4dcd31faf/kserve-container/0.log"
Sep 18 20:35:03.091033 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.090989    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"]
Sep 18 20:35:03.091332 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.091291    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" containerID="cri-o://164ab6a002fea01ee0aa2fb608d092d4cc4adc8e5711e5eb62bf055be517f56b" gracePeriod=30
Sep 18 20:35:03.091437 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.091322    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy" containerID="cri-o://008c9c08d20a99adfd6f871512f144b6178e55864b4d09f278a65f4e54046ad0" gracePeriod=30
Sep 18 20:35:03.091437 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.091322    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" containerID="cri-o://c459144f8439d5c16ae97f0b6effff579b03d8dfde89aacaa596c7e6c3f2c99c" gracePeriod=30
Sep 18 20:35:03.121586 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.121555    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"]
Sep 18 20:35:03.121819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.121797    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5" podUID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerName="kserve-container" containerID="cri-o://d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f" gracePeriod=30
Sep 18 20:35:03.121895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.121839    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5" podUID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerName="kube-rbac-proxy" containerID="cri-o://3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0" gracePeriod=30
Sep 18 20:35:03.388778 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.388754    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:35:03.522986 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.522960    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-proxy-tls\") pod \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") "
Sep 18 20:35:03.523147 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.523028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-rbac-proxy-sar-config\") pod \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") "
Sep 18 20:35:03.523212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.523189    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jswpt\" (UniqueName: \"kubernetes.io/projected/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-api-access-jswpt\") pod \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\" (UID: \"ff437fe3-a453-4c05-a70b-d3a4dcd31faf\") "
Sep 18 20:35:03.523354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.523332    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff437fe3-a453-4c05-a70b-d3a4dcd31faf" (UID: "ff437fe3-a453-4c05-a70b-d3a4dcd31faf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:03.523499 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.523482    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:35:03.525069 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.525043    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff437fe3-a453-4c05-a70b-d3a4dcd31faf" (UID: "ff437fe3-a453-4c05-a70b-d3a4dcd31faf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:35:03.525163 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.525140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-api-access-jswpt" (OuterVolumeSpecName: "kube-api-access-jswpt") pod "ff437fe3-a453-4c05-a70b-d3a4dcd31faf" (UID: "ff437fe3-a453-4c05-a70b-d3a4dcd31faf"). InnerVolumeSpecName "kube-api-access-jswpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:35:03.624142 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.624115    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswpt\" (UniqueName: \"kubernetes.io/projected/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-kube-api-access-jswpt\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:35:03.624249 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:03.624155    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff437fe3-a453-4c05-a70b-d3a4dcd31faf-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:35:04.021634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.021603    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerID="008c9c08d20a99adfd6f871512f144b6178e55864b4d09f278a65f4e54046ad0" exitCode=2
Sep 18 20:35:04.022037 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.021681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerDied","Data":"008c9c08d20a99adfd6f871512f144b6178e55864b4d09f278a65f4e54046ad0"}
Sep 18 20:35:04.023144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.023122    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerID="3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0" exitCode=2
Sep 18 20:35:04.023144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.023142    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerID="d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f" exitCode=2
Sep 18 20:35:04.023320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.023182    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"
Sep 18 20:35:04.023320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.023195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5" event={"ID":"ff437fe3-a453-4c05-a70b-d3a4dcd31faf","Type":"ContainerDied","Data":"3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0"}
Sep 18 20:35:04.023320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.023241    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5" event={"ID":"ff437fe3-a453-4c05-a70b-d3a4dcd31faf","Type":"ContainerDied","Data":"d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f"}
Sep 18 20:35:04.023320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.023261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5" event={"ID":"ff437fe3-a453-4c05-a70b-d3a4dcd31faf","Type":"ContainerDied","Data":"49d00c8d1532f9919d74f28f3cd35497542979eb39e4c2bb04a9563937c7c297"}
Sep 18 20:35:04.023320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.023284    2566 scope.go:117] "RemoveContainer" containerID="3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0"
Sep 18 20:35:04.031259 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.031244    2566 scope.go:117] "RemoveContainer" containerID="d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f"
Sep 18 20:35:04.038834 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.038815    2566 scope.go:117] "RemoveContainer" containerID="3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0"
Sep 18 20:35:04.039112 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:35:04.039095    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0\": container with ID starting with 3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0 not found: ID does not exist" containerID="3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0"
Sep 18 20:35:04.039163 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.039121    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0"} err="failed to get container status \"3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0\": rpc error: code = NotFound desc = could not find container \"3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0\": container with ID starting with 3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0 not found: ID does not exist"
Sep 18 20:35:04.039163 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.039137    2566 scope.go:117] "RemoveContainer" containerID="d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f"
Sep 18 20:35:04.039373 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:35:04.039357    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f\": container with ID starting with d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f not found: ID does not exist" containerID="d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f"
Sep 18 20:35:04.039409 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.039379    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f"} err="failed to get container status \"d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f\": rpc error: code = NotFound desc = could not find container \"d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f\": container with ID starting with d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f not found: ID does not exist"
Sep 18 20:35:04.039409 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.039394    2566 scope.go:117] "RemoveContainer" containerID="3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0"
Sep 18 20:35:04.039652 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.039632    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0"} err="failed to get container status \"3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0\": rpc error: code = NotFound desc = could not find container \"3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0\": container with ID starting with 3e2195e4ba337dab9efcc1131375bfd64c7461b7755db5adbda2881b69a1dce0 not found: ID does not exist"
Sep 18 20:35:04.039685 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.039654    2566 scope.go:117] "RemoveContainer" containerID="d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f"
Sep 18 20:35:04.039872 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.039854    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f"} err="failed to get container status \"d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f\": rpc error: code = NotFound desc = could not find container \"d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f\": container with ID starting with d824503aed3085ff870a34c3940097ca853eafdb9574c5a52a5f6eaf7cffeb3f not found: ID does not exist"
Sep 18 20:35:04.045484 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.045464    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"]
Sep 18 20:35:04.049757 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:04.049736    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-68mb5"]
Sep 18 20:35:05.459506 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:05.459460    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" path="/var/lib/kubelet/pods/ff437fe3-a453-4c05-a70b-d3a4dcd31faf/volumes"
Sep 18 20:35:05.678491 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:05.678445    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 18 20:35:08.040952 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:08.040921    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerID="164ab6a002fea01ee0aa2fb608d092d4cc4adc8e5711e5eb62bf055be517f56b" exitCode=0
Sep 18 20:35:08.041330 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:08.040968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerDied","Data":"164ab6a002fea01ee0aa2fb608d092d4cc4adc8e5711e5eb62bf055be517f56b"}
Sep 18 20:35:10.678463 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:10.678414    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 18 20:35:10.683946 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:10.683919    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:35:10.684327 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:10.684299    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:15.678311 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:15.678267    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 18 20:35:15.678729 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:15.678395    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:35:20.678492 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:20.678440    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 18 20:35:20.683899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:20.683860    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:35:20.684253 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:20.684229    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:25.678355 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:25.678271    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 18 20:35:28.436727 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.436691    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"]
Sep 18 20:35:28.437134 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437120    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy"
Sep 18 20:35:28.437183 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437137    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy"
Sep 18 20:35:28.437183 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437173    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container"
Sep 18 20:35:28.437183 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437183    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container"
Sep 18 20:35:28.437308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437199    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerName="kube-rbac-proxy"
Sep 18 20:35:28.437308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437208    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerName="kube-rbac-proxy"
Sep 18 20:35:28.437308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437219    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent"
Sep 18 20:35:28.437308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437227    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent"
Sep 18 20:35:28.437308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437238    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerName="kserve-container"
Sep 18 20:35:28.437308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437246    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerName="kserve-container"
Sep 18 20:35:28.437308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437259    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="storage-initializer"
Sep 18 20:35:28.437308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437267    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="storage-initializer"
Sep 18 20:35:28.437560 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437328    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerName="kube-rbac-proxy"
Sep 18 20:35:28.437560 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437338    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kube-rbac-proxy"
Sep 18 20:35:28.437560 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437349    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="kserve-container"
Sep 18 20:35:28.437560 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437359    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9d7e0a-7a45-416e-a5e2-e06b3e3d8db3" containerName="agent"
Sep 18 20:35:28.437560 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.437367    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff437fe3-a453-4c05-a70b-d3a4dcd31faf" containerName="kserve-container"
Sep 18 20:35:28.440703 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.440683    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.443147 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.443123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 18 20:35:28.443409 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.443392    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 18 20:35:28.449289 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.449269    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"]
Sep 18 20:35:28.523962 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.523933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d677889-be87-45ce-b01e-336a08c35ee3-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.524130 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.523972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.524130 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.524068    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.524234 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.524146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d677889-be87-45ce-b01e-336a08c35ee3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.524234 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.524198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msf2n\" (UniqueName: \"kubernetes.io/projected/7d677889-be87-45ce-b01e-336a08c35ee3-kube-api-access-msf2n\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.624791 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.624762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d677889-be87-45ce-b01e-336a08c35ee3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.627040 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.624800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msf2n\" (UniqueName: \"kubernetes.io/projected/7d677889-be87-45ce-b01e-336a08c35ee3-kube-api-access-msf2n\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.627040 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.625317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d677889-be87-45ce-b01e-336a08c35ee3-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.627040 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.625370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.627040 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.625492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.627040 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.625776    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d677889-be87-45ce-b01e-336a08c35ee3-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.627040 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.626273    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.627040 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.626395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.631463 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.631436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d677889-be87-45ce-b01e-336a08c35ee3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.633578 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.633543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msf2n\" (UniqueName: \"kubernetes.io/projected/7d677889-be87-45ce-b01e-336a08c35ee3-kube-api-access-msf2n\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:28.751909 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:28.751835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:29.021895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:29.021834    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"]
Sep 18 20:35:29.024117 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:35:29.024088    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d677889_be87_45ce_b01e_336a08c35ee3.slice/crio-097cdd84bcd3b31da33f0c90420b0de5305dea36f5565fce84c7fea01c3aec9c WatchSource:0}: Error finding container 097cdd84bcd3b31da33f0c90420b0de5305dea36f5565fce84c7fea01c3aec9c: Status 404 returned error can't find the container with id 097cdd84bcd3b31da33f0c90420b0de5305dea36f5565fce84c7fea01c3aec9c
Sep 18 20:35:29.122615 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:29.122582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" event={"ID":"7d677889-be87-45ce-b01e-336a08c35ee3","Type":"ContainerStarted","Data":"f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4"}
Sep 18 20:35:29.122751 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:29.122642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" event={"ID":"7d677889-be87-45ce-b01e-336a08c35ee3","Type":"ContainerStarted","Data":"097cdd84bcd3b31da33f0c90420b0de5305dea36f5565fce84c7fea01c3aec9c"}
Sep 18 20:35:30.678480 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:30.678441    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 18 20:35:30.683869 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:30.683833    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 20:35:30.683993 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:30.683962    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:35:30.684223 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:30.684194    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:35:30.684319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:30.684306    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:35:31.395038 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:31.394996    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:35:31.397596 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:31.397576    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:35:33.141274 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.141246    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d677889-be87-45ce-b01e-336a08c35ee3" containerID="f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4" exitCode=0
Sep 18 20:35:33.141586 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.141322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" event={"ID":"7d677889-be87-45ce-b01e-336a08c35ee3","Type":"ContainerDied","Data":"f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4"}
Sep 18 20:35:33.144213 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.144193    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerID="c459144f8439d5c16ae97f0b6effff579b03d8dfde89aacaa596c7e6c3f2c99c" exitCode=0
Sep 18 20:35:33.144302 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.144212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerDied","Data":"c459144f8439d5c16ae97f0b6effff579b03d8dfde89aacaa596c7e6c3f2c99c"}
Sep 18 20:35:33.236622 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.236597    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:35:33.259983 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.259957    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kserve-provision-location\") pod \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") "
Sep 18 20:35:33.260114 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260005    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-rbac-proxy-sar-config\") pod \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") "
Sep 18 20:35:33.260114 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260038    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-cabundle-cert\") pod \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") "
Sep 18 20:35:33.260114 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260066    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cctxc\" (UniqueName: \"kubernetes.io/projected/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-api-access-cctxc\") pod \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") "
Sep 18 20:35:33.260284 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260190    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d35638c-5ab8-4d2c-a9bc-99da6989727c-proxy-tls\") pod \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\" (UID: \"7d35638c-5ab8-4d2c-a9bc-99da6989727c\") "
Sep 18 20:35:33.260361 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260334    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d35638c-5ab8-4d2c-a9bc-99da6989727c" (UID: "7d35638c-5ab8-4d2c-a9bc-99da6989727c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:35:33.260421 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260393    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d35638c-5ab8-4d2c-a9bc-99da6989727c" (UID: "7d35638c-5ab8-4d2c-a9bc-99da6989727c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:33.260421 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d35638c-5ab8-4d2c-a9bc-99da6989727c" (UID: "7d35638c-5ab8-4d2c-a9bc-99da6989727c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:35:33.260540 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260521    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:35:33.260596 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260547    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:35:33.260596 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.260560    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d35638c-5ab8-4d2c-a9bc-99da6989727c-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:35:33.262461 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.262435    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d35638c-5ab8-4d2c-a9bc-99da6989727c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d35638c-5ab8-4d2c-a9bc-99da6989727c" (UID: "7d35638c-5ab8-4d2c-a9bc-99da6989727c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:35:33.262829 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.262435    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-api-access-cctxc" (OuterVolumeSpecName: "kube-api-access-cctxc") pod "7d35638c-5ab8-4d2c-a9bc-99da6989727c" (UID: "7d35638c-5ab8-4d2c-a9bc-99da6989727c"). InnerVolumeSpecName "kube-api-access-cctxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:35:33.361483 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.361407    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cctxc\" (UniqueName: \"kubernetes.io/projected/7d35638c-5ab8-4d2c-a9bc-99da6989727c-kube-api-access-cctxc\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:35:33.361483 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:33.361433    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d35638c-5ab8-4d2c-a9bc-99da6989727c-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:35:34.160233 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:34.160206    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"
Sep 18 20:35:34.160784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:34.160160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz" event={"ID":"7d35638c-5ab8-4d2c-a9bc-99da6989727c","Type":"ContainerDied","Data":"67cbb0dc949190aabbe62811757df1e378b66f20295304b0dd82806d12db7a76"}
Sep 18 20:35:34.160784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:34.160356    2566 scope.go:117] "RemoveContainer" containerID="c459144f8439d5c16ae97f0b6effff579b03d8dfde89aacaa596c7e6c3f2c99c"
Sep 18 20:35:34.173299 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:34.173210    2566 scope.go:117] "RemoveContainer" containerID="008c9c08d20a99adfd6f871512f144b6178e55864b4d09f278a65f4e54046ad0"
Sep 18 20:35:34.183089 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:34.183037    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"]
Sep 18 20:35:34.185665 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:34.185568    2566 scope.go:117] "RemoveContainer" containerID="164ab6a002fea01ee0aa2fb608d092d4cc4adc8e5711e5eb62bf055be517f56b"
Sep 18 20:35:34.197204 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:34.197095    2566 scope.go:117] "RemoveContainer" containerID="17a3ebcb02f1f256c93a365fab1b2629d0404694dcfb0958f88c42c8feb50856"
Sep 18 20:35:34.283563 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:34.283500    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76f676cbf7-clkgz"]
Sep 18 20:35:35.464044 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:35.463485    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" path="/var/lib/kubelet/pods/7d35638c-5ab8-4d2c-a9bc-99da6989727c/volumes"
Sep 18 20:35:55.265913 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:55.265874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" event={"ID":"7d677889-be87-45ce-b01e-336a08c35ee3","Type":"ContainerStarted","Data":"418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e"}
Sep 18 20:35:55.266396 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:55.265939    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" event={"ID":"7d677889-be87-45ce-b01e-336a08c35ee3","Type":"ContainerStarted","Data":"9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a"}
Sep 18 20:35:55.266396 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:55.266255    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:35:55.285724 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:55.285662    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" podStartSLOduration=1.577656598 podStartE2EDuration="27.28564294s" totalImagesPullingTime="21.713194649s" totalInitContainerRuntime="3.994791693s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:35:55.28516605 +0000 UTC m=+924.316092192" watchObservedRunningTime="2026-09-18 20:35:55.28564294 +0000 UTC m=+924.316569082"
Sep 18 20:35:56.270625 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:35:56.270597    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:36:02.280503 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:36:02.280476    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:36:32.281614 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:36:32.281574    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 20:36:42.280974 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:36:42.280935    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 20:36:52.280996 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:36:52.280960    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 20:37:02.281976 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:02.281895    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 20:37:11.969321 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:11.969284    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:37:18.568868 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:18.568838    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"]
Sep 18 20:37:18.569400 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:18.569079    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kserve-container" containerID="cri-o://9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a" gracePeriod=30
Sep 18 20:37:18.569400 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:18.569144    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kube-rbac-proxy" containerID="cri-o://418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e" gracePeriod=30
Sep 18 20:37:19.586584 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:19.586546    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d677889-be87-45ce-b01e-336a08c35ee3" containerID="418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e" exitCode=2
Sep 18 20:37:19.586584 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:19.586586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" event={"ID":"7d677889-be87-45ce-b01e-336a08c35ee3","Type":"ContainerDied","Data":"418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e"}
Sep 18 20:37:21.459203 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:21.459158    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 20:37:22.275616 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:22.275572    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 18 20:37:23.409963 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.409934    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:37:23.539892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.539816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d677889-be87-45ce-b01e-336a08c35ee3-kserve-provision-location\") pod \"7d677889-be87-45ce-b01e-336a08c35ee3\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") "
Sep 18 20:37:23.539892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.539870    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-kube-rbac-proxy-sar-config\") pod \"7d677889-be87-45ce-b01e-336a08c35ee3\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") "
Sep 18 20:37:23.540148 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.539912    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d677889-be87-45ce-b01e-336a08c35ee3-proxy-tls\") pod \"7d677889-be87-45ce-b01e-336a08c35ee3\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") "
Sep 18 20:37:23.540148 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.539958    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-cabundle-cert\") pod \"7d677889-be87-45ce-b01e-336a08c35ee3\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") "
Sep 18 20:37:23.540148 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.539982    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msf2n\" (UniqueName: \"kubernetes.io/projected/7d677889-be87-45ce-b01e-336a08c35ee3-kube-api-access-msf2n\") pod \"7d677889-be87-45ce-b01e-336a08c35ee3\" (UID: \"7d677889-be87-45ce-b01e-336a08c35ee3\") "
Sep 18 20:37:23.540308 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.540163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d677889-be87-45ce-b01e-336a08c35ee3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d677889-be87-45ce-b01e-336a08c35ee3" (UID: "7d677889-be87-45ce-b01e-336a08c35ee3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:23.540365 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.540325    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d677889-be87-45ce-b01e-336a08c35ee3" (UID: "7d677889-be87-45ce-b01e-336a08c35ee3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:37:23.540365 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.540332    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d677889-be87-45ce-b01e-336a08c35ee3" (UID: "7d677889-be87-45ce-b01e-336a08c35ee3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:37:23.542071 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.542047    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d677889-be87-45ce-b01e-336a08c35ee3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d677889-be87-45ce-b01e-336a08c35ee3" (UID: "7d677889-be87-45ce-b01e-336a08c35ee3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:37:23.542271 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.542243    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d677889-be87-45ce-b01e-336a08c35ee3-kube-api-access-msf2n" (OuterVolumeSpecName: "kube-api-access-msf2n") pod "7d677889-be87-45ce-b01e-336a08c35ee3" (UID: "7d677889-be87-45ce-b01e-336a08c35ee3"). InnerVolumeSpecName "kube-api-access-msf2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:37:23.607921 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.607877    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d677889-be87-45ce-b01e-336a08c35ee3" containerID="9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a" exitCode=0
Sep 18 20:37:23.608106 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.607929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" event={"ID":"7d677889-be87-45ce-b01e-336a08c35ee3","Type":"ContainerDied","Data":"9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a"}
Sep 18 20:37:23.608106 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.607959    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"
Sep 18 20:37:23.608106 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.607977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b" event={"ID":"7d677889-be87-45ce-b01e-336a08c35ee3","Type":"ContainerDied","Data":"097cdd84bcd3b31da33f0c90420b0de5305dea36f5565fce84c7fea01c3aec9c"}
Sep 18 20:37:23.608106 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.608002    2566 scope.go:117] "RemoveContainer" containerID="418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e"
Sep 18 20:37:23.618424 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.618404    2566 scope.go:117] "RemoveContainer" containerID="9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a"
Sep 18 20:37:23.626657 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.626641    2566 scope.go:117] "RemoveContainer" containerID="f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4"
Sep 18 20:37:23.633142 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.633119    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"]
Sep 18 20:37:23.634647 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.634634    2566 scope.go:117] "RemoveContainer" containerID="418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e"
Sep 18 20:37:23.634932 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:37:23.634906    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e\": container with ID starting with 418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e not found: ID does not exist" containerID="418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e"
Sep 18 20:37:23.635000 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.634938    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e"} err="failed to get container status \"418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e\": rpc error: code = NotFound desc = could not find container \"418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e\": container with ID starting with 418e8fe3648ea3d0cbf6bc332e5868606fee36ae8bb84d1cb99e49be096a376e not found: ID does not exist"
Sep 18 20:37:23.635000 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.634959    2566 scope.go:117] "RemoveContainer" containerID="9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a"
Sep 18 20:37:23.635245 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:37:23.635230    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a\": container with ID starting with 9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a not found: ID does not exist" containerID="9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a"
Sep 18 20:37:23.635287 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.635251    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a"} err="failed to get container status \"9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a\": rpc error: code = NotFound desc = could not find container \"9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a\": container with ID starting with 9bf0ad674a1f003237605edd586569a876b840d6385fb3f3a1de40e933b6bd6a not found: ID does not exist"
Sep 18 20:37:23.635287 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.635269    2566 scope.go:117] "RemoveContainer" containerID="f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4"
Sep 18 20:37:23.635503 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:37:23.635489    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4\": container with ID starting with f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4 not found: ID does not exist" containerID="f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4"
Sep 18 20:37:23.635554 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.635505    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4"} err="failed to get container status \"f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4\": rpc error: code = NotFound desc = could not find container \"f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4\": container with ID starting with f3147b17fc5a08b82e903587c1ae7e8fe0f9d34134daffc83986f45a06a780d4 not found: ID does not exist"
Sep 18 20:37:23.638947 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.638923    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-p6n5b"]
Sep 18 20:37:23.640863 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.640841    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d677889-be87-45ce-b01e-336a08c35ee3-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:37:23.640956 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.640867    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:37:23.640956 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.640881    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d677889-be87-45ce-b01e-336a08c35ee3-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:37:23.640956 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.640906    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d677889-be87-45ce-b01e-336a08c35ee3-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:37:23.640956 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.640919    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msf2n\" (UniqueName: \"kubernetes.io/projected/7d677889-be87-45ce-b01e-336a08c35ee3-kube-api-access-msf2n\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:37:23.757786 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.757757    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"]
Sep 18 20:37:23.758113 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758097    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kube-rbac-proxy"
Sep 18 20:37:23.758212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758115    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kube-rbac-proxy"
Sep 18 20:37:23.758212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758131    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container"
Sep 18 20:37:23.758212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758139    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container"
Sep 18 20:37:23.758212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758155    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kserve-container"
Sep 18 20:37:23.758212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758164    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kserve-container"
Sep 18 20:37:23.758212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758175    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy"
Sep 18 20:37:23.758212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758185    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy"
Sep 18 20:37:23.758212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758196    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent"
Sep 18 20:37:23.758212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758205    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent"
Sep 18 20:37:23.758661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758216    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="storage-initializer"
Sep 18 20:37:23.758661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758225    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="storage-initializer"
Sep 18 20:37:23.758661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758243    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="storage-initializer"
Sep 18 20:37:23.758661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758251    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="storage-initializer"
Sep 18 20:37:23.758661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758321    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kube-rbac-proxy"
Sep 18 20:37:23.758661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758331    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kserve-container"
Sep 18 20:37:23.758661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758342    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="agent"
Sep 18 20:37:23.758661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758353    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" containerName="kube-rbac-proxy"
Sep 18 20:37:23.758661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.758363    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d35638c-5ab8-4d2c-a9bc-99da6989727c" containerName="kserve-container"
Sep 18 20:37:23.763143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.763123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:23.765952 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.765930    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 18 20:37:23.766121 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.766100    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:37:23.766391 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.766173    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:37:23.766391 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.766182    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 18 20:37:23.766391 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.766213    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:37:23.766391 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.766214    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:37:23.766391 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.766263    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:37:23.771458 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.771431    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"]
Sep 18 20:37:23.944004 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.943915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a79ae3-3529-462a-89f6-a3a2f63d627b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:23.944004 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.943973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:23.944233 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.944078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:23.944233 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.944115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fxs\" (UniqueName: \"kubernetes.io/projected/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-api-access-h8fxs\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:23.944233 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:23.944144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a79ae3-3529-462a-89f6-a3a2f63d627b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.045351 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.045317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a79ae3-3529-462a-89f6-a3a2f63d627b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.045351 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.045368    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a79ae3-3529-462a-89f6-a3a2f63d627b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.045590 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.045393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.045590 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.045426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.045590 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.045446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fxs\" (UniqueName: \"kubernetes.io/projected/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-api-access-h8fxs\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.045830 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.045805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a79ae3-3529-462a-89f6-a3a2f63d627b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.046158 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.046136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.046242 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.046171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.047851 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.047828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a79ae3-3529-462a-89f6-a3a2f63d627b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.054372 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.054353    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fxs\" (UniqueName: \"kubernetes.io/projected/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-api-access-h8fxs\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.074227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.074205    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:24.343992 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.343767    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"]
Sep 18 20:37:24.346743 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:37:24.346717    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a79ae3_3529_462a_89f6_a3a2f63d627b.slice/crio-96b45086ac9718dc324a1493d6740de47e7eccdf97a607a7da5b595993fe067a WatchSource:0}: Error finding container 96b45086ac9718dc324a1493d6740de47e7eccdf97a607a7da5b595993fe067a: Status 404 returned error can't find the container with id 96b45086ac9718dc324a1493d6740de47e7eccdf97a607a7da5b595993fe067a
Sep 18 20:37:24.615306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.615221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" event={"ID":"03a79ae3-3529-462a-89f6-a3a2f63d627b","Type":"ContainerStarted","Data":"f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98"}
Sep 18 20:37:24.615306 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:24.615283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" event={"ID":"03a79ae3-3529-462a-89f6-a3a2f63d627b","Type":"ContainerStarted","Data":"96b45086ac9718dc324a1493d6740de47e7eccdf97a607a7da5b595993fe067a"}
Sep 18 20:37:25.460901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:25.460858    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d677889-be87-45ce-b01e-336a08c35ee3" path="/var/lib/kubelet/pods/7d677889-be87-45ce-b01e-336a08c35ee3/volumes"
Sep 18 20:37:28.636908 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:28.636874    2566 generic.go:356] "Generic (PLEG): container finished" podID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerID="f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98" exitCode=0
Sep 18 20:37:28.637274 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:28.636933    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" event={"ID":"03a79ae3-3529-462a-89f6-a3a2f63d627b","Type":"ContainerDied","Data":"f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98"}
Sep 18 20:37:29.642545 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:29.642511    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" event={"ID":"03a79ae3-3529-462a-89f6-a3a2f63d627b","Type":"ContainerStarted","Data":"293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e"}
Sep 18 20:37:29.642954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:29.642577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" event={"ID":"03a79ae3-3529-462a-89f6-a3a2f63d627b","Type":"ContainerStarted","Data":"35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa"}
Sep 18 20:37:29.642954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:29.642772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:29.664212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:29.664166    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" podStartSLOduration=2.847654857 podStartE2EDuration="6.664151484s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.816496627s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:37:29.662677592 +0000 UTC m=+1018.693603733" watchObservedRunningTime="2026-09-18 20:37:29.664151484 +0000 UTC m=+1018.695077633"
Sep 18 20:37:30.647089 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:30.647058    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:37:36.656443 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:37:36.656417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:38:06.658228 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:06.658185    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:38:16.656984 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:16.656941    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:38:26.657942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:26.657862    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:38:36.657600 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:36.657556    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:38:44.968448 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:44.968415    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:38:53.796222 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:53.796183    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"]
Sep 18 20:38:53.796645 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:53.796408    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kserve-container" containerID="cri-o://35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa" gracePeriod=30
Sep 18 20:38:53.796645 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:53.796444    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kube-rbac-proxy" containerID="cri-o://293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e" gracePeriod=30
Sep 18 20:38:53.962946 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:53.962916    2566 generic.go:356] "Generic (PLEG): container finished" podID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerID="293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e" exitCode=2
Sep 18 20:38:53.963074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:53.962957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" event={"ID":"03a79ae3-3529-462a-89f6-a3a2f63d627b","Type":"ContainerDied","Data":"293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e"}
Sep 18 20:38:54.456199 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:54.456159    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 20:38:56.652147 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:56.652112    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 20:38:58.538327 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.538302    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:38:58.572650 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.572620    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-cabundle-cert\") pod \"03a79ae3-3529-462a-89f6-a3a2f63d627b\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") "
Sep 18 20:38:58.572771 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.572673    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8fxs\" (UniqueName: \"kubernetes.io/projected/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-api-access-h8fxs\") pod \"03a79ae3-3529-462a-89f6-a3a2f63d627b\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") "
Sep 18 20:38:58.572771 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.572765    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a79ae3-3529-462a-89f6-a3a2f63d627b-proxy-tls\") pod \"03a79ae3-3529-462a-89f6-a3a2f63d627b\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") "
Sep 18 20:38:58.572874 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.572802    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-rbac-proxy-sar-config\") pod \"03a79ae3-3529-462a-89f6-a3a2f63d627b\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") "
Sep 18 20:38:58.572874 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.572855    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a79ae3-3529-462a-89f6-a3a2f63d627b-kserve-provision-location\") pod \"03a79ae3-3529-462a-89f6-a3a2f63d627b\" (UID: \"03a79ae3-3529-462a-89f6-a3a2f63d627b\") "
Sep 18 20:38:58.573131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.572960    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03a79ae3-3529-462a-89f6-a3a2f63d627b" (UID: "03a79ae3-3529-462a-89f6-a3a2f63d627b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:38:58.573131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.573080    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:38:58.573272 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.573149    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a79ae3-3529-462a-89f6-a3a2f63d627b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03a79ae3-3529-462a-89f6-a3a2f63d627b" (UID: "03a79ae3-3529-462a-89f6-a3a2f63d627b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:38:58.573272 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.573170    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03a79ae3-3529-462a-89f6-a3a2f63d627b" (UID: "03a79ae3-3529-462a-89f6-a3a2f63d627b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:38:58.574793 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.574774    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a79ae3-3529-462a-89f6-a3a2f63d627b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03a79ae3-3529-462a-89f6-a3a2f63d627b" (UID: "03a79ae3-3529-462a-89f6-a3a2f63d627b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:38:58.574865 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.574817    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-api-access-h8fxs" (OuterVolumeSpecName: "kube-api-access-h8fxs") pod "03a79ae3-3529-462a-89f6-a3a2f63d627b" (UID: "03a79ae3-3529-462a-89f6-a3a2f63d627b"). InnerVolumeSpecName "kube-api-access-h8fxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:38:58.674386 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.674299    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a79ae3-3529-462a-89f6-a3a2f63d627b-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:38:58.674386 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.674339    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:38:58.674386 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.674352    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a79ae3-3529-462a-89f6-a3a2f63d627b-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:38:58.674386 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.674364    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8fxs\" (UniqueName: \"kubernetes.io/projected/03a79ae3-3529-462a-89f6-a3a2f63d627b-kube-api-access-h8fxs\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:38:58.984579 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.984502    2566 generic.go:356] "Generic (PLEG): container finished" podID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerID="35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa" exitCode=0
Sep 18 20:38:58.984579 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.984566    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"
Sep 18 20:38:58.984745 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.984566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" event={"ID":"03a79ae3-3529-462a-89f6-a3a2f63d627b","Type":"ContainerDied","Data":"35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa"}
Sep 18 20:38:58.984745 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.984640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck" event={"ID":"03a79ae3-3529-462a-89f6-a3a2f63d627b","Type":"ContainerDied","Data":"96b45086ac9718dc324a1493d6740de47e7eccdf97a607a7da5b595993fe067a"}
Sep 18 20:38:58.984745 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.984680    2566 scope.go:117] "RemoveContainer" containerID="293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e"
Sep 18 20:38:58.995703 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:58.995657    2566 scope.go:117] "RemoveContainer" containerID="35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa"
Sep 18 20:38:59.000762 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.000736    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"]
Sep 18 20:38:59.001155 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.001137    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kube-rbac-proxy"
Sep 18 20:38:59.001155 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.001155    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kube-rbac-proxy"
Sep 18 20:38:59.001304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.001172    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kserve-container"
Sep 18 20:38:59.001304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.001182    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kserve-container"
Sep 18 20:38:59.001304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.001202    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="storage-initializer"
Sep 18 20:38:59.001304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.001210    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="storage-initializer"
Sep 18 20:38:59.001304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.001274    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kube-rbac-proxy"
Sep 18 20:38:59.001304 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.001288    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" containerName="kserve-container"
Sep 18 20:38:59.004039 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.004000    2566 scope.go:117] "RemoveContainer" containerID="f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98"
Sep 18 20:38:59.005217 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.005199    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.008249 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.008225    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 18 20:38:59.008361 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.008254    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:38:59.008361 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.008259    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 18 20:38:59.008361 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.008273    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:38:59.008361 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.008259    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:38:59.008584 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.008384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:38:59.008584 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.008431    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:38:59.013565 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.013505    2566 scope.go:117] "RemoveContainer" containerID="293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e"
Sep 18 20:38:59.014831 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:38:59.014683    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e\": container with ID starting with 293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e not found: ID does not exist" containerID="293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e"
Sep 18 20:38:59.014831 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.014719    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e"} err="failed to get container status \"293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e\": rpc error: code = NotFound desc = could not find container \"293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e\": container with ID starting with 293fdc9b248680777c6f7950b4afc58fbb5dedb62381ec6afeac739115d3e97e not found: ID does not exist"
Sep 18 20:38:59.014831 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.014742    2566 scope.go:117] "RemoveContainer" containerID="35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa"
Sep 18 20:38:59.015497 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:38:59.015460    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa\": container with ID starting with 35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa not found: ID does not exist" containerID="35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa"
Sep 18 20:38:59.015603 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.015493    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa"} err="failed to get container status \"35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa\": rpc error: code = NotFound desc = could not find container \"35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa\": container with ID starting with 35fdbea6d12c183b4de819c0ec9856d5c8bb5fb0e13cc62b6d3df51cebb9b8aa not found: ID does not exist"
Sep 18 20:38:59.015603 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.015517    2566 scope.go:117] "RemoveContainer" containerID="f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98"
Sep 18 20:38:59.015831 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:38:59.015809    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98\": container with ID starting with f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98 not found: ID does not exist" containerID="f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98"
Sep 18 20:38:59.015902 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.015837    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98"} err="failed to get container status \"f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98\": rpc error: code = NotFound desc = could not find container \"f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98\": container with ID starting with f692408e54df0505fefab7f1eac781186ada577899590d473f66716238e08f98 not found: ID does not exist"
Sep 18 20:38:59.016633 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.016616    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"]
Sep 18 20:38:59.018072 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.018052    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"]
Sep 18 20:38:59.024607 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.024590    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-6gtck"]
Sep 18 20:38:59.078193 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.078159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjfk4\" (UniqueName: \"kubernetes.io/projected/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-api-access-tjfk4\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.078193 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.078195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.078361 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.078217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f79c8-763d-4e19-8291-cc22f8f32372-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.078361 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.078319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f7f79c8-763d-4e19-8291-cc22f8f32372-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.078361 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.078345    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.179095 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.179068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f7f79c8-763d-4e19-8291-cc22f8f32372-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.179095 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.179097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.179313 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.179293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjfk4\" (UniqueName: \"kubernetes.io/projected/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-api-access-tjfk4\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.179372 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.179339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.179372 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.179361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f79c8-763d-4e19-8291-cc22f8f32372-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.179706 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.179686    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f79c8-763d-4e19-8291-cc22f8f32372-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.179805 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.179795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.180054 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.180007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.181616 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.181598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f7f79c8-763d-4e19-8291-cc22f8f32372-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.187352 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.187336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjfk4\" (UniqueName: \"kubernetes.io/projected/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-api-access-tjfk4\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.316847 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.316824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:38:59.459899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.459866    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a79ae3-3529-462a-89f6-a3a2f63d627b" path="/var/lib/kubelet/pods/03a79ae3-3529-462a-89f6-a3a2f63d627b/volumes"
Sep 18 20:38:59.587699 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.587551    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"]
Sep 18 20:38:59.590210 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:38:59.590184    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7f79c8_763d_4e19_8291_cc22f8f32372.slice/crio-c236b603fe9e7c64cf6a9e5839429221d51e1e59c55d1717d49c303aa0e3bbb9 WatchSource:0}: Error finding container c236b603fe9e7c64cf6a9e5839429221d51e1e59c55d1717d49c303aa0e3bbb9: Status 404 returned error can't find the container with id c236b603fe9e7c64cf6a9e5839429221d51e1e59c55d1717d49c303aa0e3bbb9
Sep 18 20:38:59.990554 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.990476    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" event={"ID":"8f7f79c8-763d-4e19-8291-cc22f8f32372","Type":"ContainerStarted","Data":"69f536acb0bdaed7c3931b8cf50a48375bd7d2f5f70067f46722292ce90a13bc"}
Sep 18 20:38:59.990554 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:38:59.990536    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" event={"ID":"8f7f79c8-763d-4e19-8291-cc22f8f32372","Type":"ContainerStarted","Data":"c236b603fe9e7c64cf6a9e5839429221d51e1e59c55d1717d49c303aa0e3bbb9"}
Sep 18 20:39:04.006336 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:04.006259    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerID="69f536acb0bdaed7c3931b8cf50a48375bd7d2f5f70067f46722292ce90a13bc" exitCode=0
Sep 18 20:39:04.006675 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:04.006329    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" event={"ID":"8f7f79c8-763d-4e19-8291-cc22f8f32372","Type":"ContainerDied","Data":"69f536acb0bdaed7c3931b8cf50a48375bd7d2f5f70067f46722292ce90a13bc"}
Sep 18 20:39:05.011818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:05.011784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" event={"ID":"8f7f79c8-763d-4e19-8291-cc22f8f32372","Type":"ContainerStarted","Data":"36022a4e6efcfbd35c49a0a8484cfee9caf608de25fe1978d5b8178c873635b2"}
Sep 18 20:39:05.012299 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:05.011854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" event={"ID":"8f7f79c8-763d-4e19-8291-cc22f8f32372","Type":"ContainerStarted","Data":"e407847f2fe16ea97d8b67e32f66d914e5508d95c4282720df478e80a0774dcc"}
Sep 18 20:39:05.012299 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:05.012238    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:39:05.012299 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:05.012284    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:39:05.032455 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:05.032407    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" podStartSLOduration=3.03780744 podStartE2EDuration="7.032393229s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.994585789s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:39:05.031167632 +0000 UTC m=+1114.062093776" watchObservedRunningTime="2026-09-18 20:39:05.032393229 +0000 UTC m=+1114.063319372"
Sep 18 20:39:11.021729 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:11.021699    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:39:41.023242 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:41.023203    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 20:39:51.022478 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:39:51.022441    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 20:40:01.023303 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:01.023212    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 20:40:11.022554 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:11.022511    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 20:40:21.568235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:21.568206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:40:29.108573 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:29.108537    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"]
Sep 18 20:40:29.108973 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:29.108831    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kserve-container" containerID="cri-o://e407847f2fe16ea97d8b67e32f66d914e5508d95c4282720df478e80a0774dcc" gracePeriod=30
Sep 18 20:40:29.108973 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:29.108847    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kube-rbac-proxy" containerID="cri-o://36022a4e6efcfbd35c49a0a8484cfee9caf608de25fe1978d5b8178c873635b2" gracePeriod=30
Sep 18 20:40:29.331297 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:29.331256    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerID="36022a4e6efcfbd35c49a0a8484cfee9caf608de25fe1978d5b8178c873635b2" exitCode=2
Sep 18 20:40:29.331468 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:29.331330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" event={"ID":"8f7f79c8-763d-4e19-8291-cc22f8f32372","Type":"ContainerDied","Data":"36022a4e6efcfbd35c49a0a8484cfee9caf608de25fe1978d5b8178c873635b2"}
Sep 18 20:40:31.016525 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:31.016482    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 18 20:40:31.420102 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:31.420076    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:40:31.423368 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:31.423347    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:40:31.568796 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:31.568752    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 20:40:34.352885 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.352854    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerID="e407847f2fe16ea97d8b67e32f66d914e5508d95c4282720df478e80a0774dcc" exitCode=0
Sep 18 20:40:34.353301 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.352934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" event={"ID":"8f7f79c8-763d-4e19-8291-cc22f8f32372","Type":"ContainerDied","Data":"e407847f2fe16ea97d8b67e32f66d914e5508d95c4282720df478e80a0774dcc"}
Sep 18 20:40:34.544185 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.544165    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:40:34.621497 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.621426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-cabundle-cert\") pod \"8f7f79c8-763d-4e19-8291-cc22f8f32372\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") "
Sep 18 20:40:34.621497 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.621468    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f7f79c8-763d-4e19-8291-cc22f8f32372-proxy-tls\") pod \"8f7f79c8-763d-4e19-8291-cc22f8f32372\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") "
Sep 18 20:40:34.621497 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.621489    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjfk4\" (UniqueName: \"kubernetes.io/projected/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-api-access-tjfk4\") pod \"8f7f79c8-763d-4e19-8291-cc22f8f32372\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") "
Sep 18 20:40:34.621754 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.621518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f79c8-763d-4e19-8291-cc22f8f32372-kserve-provision-location\") pod \"8f7f79c8-763d-4e19-8291-cc22f8f32372\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") "
Sep 18 20:40:34.621754 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.621551    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-rbac-proxy-sar-config\") pod \"8f7f79c8-763d-4e19-8291-cc22f8f32372\" (UID: \"8f7f79c8-763d-4e19-8291-cc22f8f32372\") "
Sep 18 20:40:34.621863 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.621832    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7f79c8-763d-4e19-8291-cc22f8f32372-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f7f79c8-763d-4e19-8291-cc22f8f32372" (UID: "8f7f79c8-763d-4e19-8291-cc22f8f32372"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:40:34.621941 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.621914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f7f79c8-763d-4e19-8291-cc22f8f32372" (UID: "8f7f79c8-763d-4e19-8291-cc22f8f32372"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:40:34.622000 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.621980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f7f79c8-763d-4e19-8291-cc22f8f32372" (UID: "8f7f79c8-763d-4e19-8291-cc22f8f32372"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:40:34.623544 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.623524    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-api-access-tjfk4" (OuterVolumeSpecName: "kube-api-access-tjfk4") pod "8f7f79c8-763d-4e19-8291-cc22f8f32372" (UID: "8f7f79c8-763d-4e19-8291-cc22f8f32372"). InnerVolumeSpecName "kube-api-access-tjfk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:40:34.623624 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.623603    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7f79c8-763d-4e19-8291-cc22f8f32372-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f7f79c8-763d-4e19-8291-cc22f8f32372" (UID: "8f7f79c8-763d-4e19-8291-cc22f8f32372"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:40:34.722239 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.722212    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f7f79c8-763d-4e19-8291-cc22f8f32372-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:40:34.722239 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.722234    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:40:34.722239 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.722244    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f7f79c8-763d-4e19-8291-cc22f8f32372-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:40:34.722481 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.722252    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f7f79c8-763d-4e19-8291-cc22f8f32372-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:40:34.722481 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:34.722260    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjfk4\" (UniqueName: \"kubernetes.io/projected/8f7f79c8-763d-4e19-8291-cc22f8f32372-kube-api-access-tjfk4\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:40:35.358092 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:35.358054    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d" event={"ID":"8f7f79c8-763d-4e19-8291-cc22f8f32372","Type":"ContainerDied","Data":"c236b603fe9e7c64cf6a9e5839429221d51e1e59c55d1717d49c303aa0e3bbb9"}
Sep 18 20:40:35.358469 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:35.358135    2566 scope.go:117] "RemoveContainer" containerID="36022a4e6efcfbd35c49a0a8484cfee9caf608de25fe1978d5b8178c873635b2"
Sep 18 20:40:35.358469 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:35.358168    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"
Sep 18 20:40:35.366448 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:35.366345    2566 scope.go:117] "RemoveContainer" containerID="e407847f2fe16ea97d8b67e32f66d914e5508d95c4282720df478e80a0774dcc"
Sep 18 20:40:35.373645 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:35.373629    2566 scope.go:117] "RemoveContainer" containerID="69f536acb0bdaed7c3931b8cf50a48375bd7d2f5f70067f46722292ce90a13bc"
Sep 18 20:40:35.380480 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:35.380458    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"]
Sep 18 20:40:35.383296 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:35.383274    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-qzq5d"]
Sep 18 20:40:35.459492 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:35.459467    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" path="/var/lib/kubelet/pods/8f7f79c8-763d-4e19-8291-cc22f8f32372/volumes"
Sep 18 20:40:36.152707 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.152680    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"]
Sep 18 20:40:36.152959 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.152945    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="storage-initializer"
Sep 18 20:40:36.153032 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.152962    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="storage-initializer"
Sep 18 20:40:36.153032 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.152976    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kserve-container"
Sep 18 20:40:36.153032 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.152982    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kserve-container"
Sep 18 20:40:36.153032 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.152992    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kube-rbac-proxy"
Sep 18 20:40:36.153032 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.152997    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kube-rbac-proxy"
Sep 18 20:40:36.153199 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.153063    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kserve-container"
Sep 18 20:40:36.153199 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.153074    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7f79c8-763d-4e19-8291-cc22f8f32372" containerName="kube-rbac-proxy"
Sep 18 20:40:36.157525 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.157507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.160378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.160354    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:40:36.160577 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.160554    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 18 20:40:36.160694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.160594    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:40:36.160694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.160607    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:40:36.160694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.160563    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:40:36.160694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.160591    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 18 20:40:36.161472 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.161459    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:40:36.167521 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.167498    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"]
Sep 18 20:40:36.231726 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.231699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.231880 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.231742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsng8\" (UniqueName: \"kubernetes.io/projected/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-api-access-hsng8\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.231880 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.231773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.231880 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.231854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273fdfa6-fa4d-4a55-83eb-21955d1b1240-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.231982 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.231902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.332979 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.332945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273fdfa6-fa4d-4a55-83eb-21955d1b1240-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.333154 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.332994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.333154 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.333050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.333154 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.333096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsng8\" (UniqueName: \"kubernetes.io/projected/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-api-access-hsng8\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.333154 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.333129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.333560 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.333537    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.333749 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.333731    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.333814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.333791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.335330 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.335312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273fdfa6-fa4d-4a55-83eb-21955d1b1240-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.341578 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.341554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsng8\" (UniqueName: \"kubernetes.io/projected/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-api-access-hsng8\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.469492 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.469419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:40:36.741283 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:36.741256    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"]
Sep 18 20:40:36.743990 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:40:36.743958    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273fdfa6_fa4d_4a55_83eb_21955d1b1240.slice/crio-95c6ca44e47073e388eb4ddcf04070e653c251da49aaf9c3758f59fcfa5d98cc WatchSource:0}: Error finding container 95c6ca44e47073e388eb4ddcf04070e653c251da49aaf9c3758f59fcfa5d98cc: Status 404 returned error can't find the container with id 95c6ca44e47073e388eb4ddcf04070e653c251da49aaf9c3758f59fcfa5d98cc
Sep 18 20:40:37.367581 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:37.367544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" event={"ID":"273fdfa6-fa4d-4a55-83eb-21955d1b1240","Type":"ContainerStarted","Data":"c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f"}
Sep 18 20:40:37.367779 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:37.367617    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" event={"ID":"273fdfa6-fa4d-4a55-83eb-21955d1b1240","Type":"ContainerStarted","Data":"95c6ca44e47073e388eb4ddcf04070e653c251da49aaf9c3758f59fcfa5d98cc"}
Sep 18 20:40:51.946397 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:51.946368    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-4e879ee6"]
Sep 18 20:40:51.950911 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:51.950895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-4e879ee6"
Sep 18 20:40:51.958007 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:51.957984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-4e879ee6"]
Sep 18 20:40:52.054089 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:52.054061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2fth\" (UniqueName: \"kubernetes.io/projected/e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0-kube-api-access-v2fth\") pod \"curl-metrics-4e879ee6\" (UID: \"e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0\") " pod="kserve/curl-metrics-4e879ee6"
Sep 18 20:40:52.154993 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:52.154954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2fth\" (UniqueName: \"kubernetes.io/projected/e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0-kube-api-access-v2fth\") pod \"curl-metrics-4e879ee6\" (UID: \"e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0\") " pod="kserve/curl-metrics-4e879ee6"
Sep 18 20:40:52.163432 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:52.163413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2fth\" (UniqueName: \"kubernetes.io/projected/e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0-kube-api-access-v2fth\") pod \"curl-metrics-4e879ee6\" (UID: \"e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0\") " pod="kserve/curl-metrics-4e879ee6"
Sep 18 20:40:52.273145 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:52.273109    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-4e879ee6"
Sep 18 20:40:52.542486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:52.542463    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-4e879ee6"]
Sep 18 20:40:52.544403 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:40:52.544367    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8536d0f_9cb1_45de_a9b9_1fd4da1e8cb0.slice/crio-bb5d76ce210f7cc85909118ddbe2d017ddcfe0d9b2352a19a67d47f088880bf6 WatchSource:0}: Error finding container bb5d76ce210f7cc85909118ddbe2d017ddcfe0d9b2352a19a67d47f088880bf6: Status 404 returned error can't find the container with id bb5d76ce210f7cc85909118ddbe2d017ddcfe0d9b2352a19a67d47f088880bf6
Sep 18 20:40:52.546338 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:52.546320    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:40:53.431641 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:53.431603    2566 generic.go:356] "Generic (PLEG): container finished" podID="e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0" containerID="dc5a8389f1dff6b00d7680dd3671bd8ad65d6a50627ac56ead75419cbc8b6c17" exitCode=0
Sep 18 20:40:53.432046 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:53.431683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-4e879ee6" event={"ID":"e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0","Type":"ContainerDied","Data":"dc5a8389f1dff6b00d7680dd3671bd8ad65d6a50627ac56ead75419cbc8b6c17"}
Sep 18 20:40:53.432046 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:53.431709    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-4e879ee6" event={"ID":"e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0","Type":"ContainerStarted","Data":"bb5d76ce210f7cc85909118ddbe2d017ddcfe0d9b2352a19a67d47f088880bf6"}
Sep 18 20:40:54.552214 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:54.552189    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-4e879ee6"
Sep 18 20:40:54.587192 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:54.587161    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-4e879ee6_e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0/curl-metrics-4e879ee6/0.log"
Sep 18 20:40:54.609892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:54.609866    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-4e879ee6"]
Sep 18 20:40:54.615586 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:54.615567    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-4e879ee6"]
Sep 18 20:40:54.677647 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:54.677626    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2fth\" (UniqueName: \"kubernetes.io/projected/e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0-kube-api-access-v2fth\") pod \"e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0\" (UID: \"e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0\") "
Sep 18 20:40:54.679654 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:54.679625    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0-kube-api-access-v2fth" (OuterVolumeSpecName: "kube-api-access-v2fth") pod "e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0" (UID: "e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0"). InnerVolumeSpecName "kube-api-access-v2fth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:40:54.778726 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:54.778705    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2fth\" (UniqueName: \"kubernetes.io/projected/e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0-kube-api-access-v2fth\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:40:55.440893 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:55.440865    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb5d76ce210f7cc85909118ddbe2d017ddcfe0d9b2352a19a67d47f088880bf6"
Sep 18 20:40:55.441109 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:55.440910    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-4e879ee6"
Sep 18 20:40:55.451114 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:40:55.451085    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-4e879ee6\" is forbidden: User \"system:node:ip-10-0-143-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-143-76.ec2.internal' and this object" podUID="e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0" pod="kserve/curl-metrics-4e879ee6"
Sep 18 20:40:55.459563 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:55.459539    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0" path="/var/lib/kubelet/pods/e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0/volumes"
Sep 18 20:40:57.449846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:57.449760    2566 generic.go:356] "Generic (PLEG): container finished" podID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerID="c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f" exitCode=0
Sep 18 20:40:57.450268 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:40:57.449837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" event={"ID":"273fdfa6-fa4d-4a55-83eb-21955d1b1240","Type":"ContainerDied","Data":"c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f"}
Sep 18 20:41:08.686886 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.686850    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"]
Sep 18 20:41:08.687379 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.687305    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0" containerName="curl-metrics-4e879ee6"
Sep 18 20:41:08.687379 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.687323    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0" containerName="curl-metrics-4e879ee6"
Sep 18 20:41:08.687521 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.687410    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8536d0f-9cb1-45de-a9b9-1fd4da1e8cb0" containerName="curl-metrics-4e879ee6"
Sep 18 20:41:08.690496 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.690474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.693327 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.693305    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 18 20:41:08.693754 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.693500    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 18 20:41:08.700904 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.700833    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"]
Sep 18 20:41:08.795520 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.795488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-cabundle-cert\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.795704 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.795531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4259\" (UniqueName: \"kubernetes.io/projected/12c31135-f6df-4a38-9306-13579e12ceec-kube-api-access-z4259\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.795704 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.795569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c31135-f6df-4a38-9306-13579e12ceec-kserve-provision-location\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.795704 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.795640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.795839 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.795736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c31135-f6df-4a38-9306-13579e12ceec-proxy-tls\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.896191 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.896159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.896371 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.896198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c31135-f6df-4a38-9306-13579e12ceec-proxy-tls\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.896371 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.896225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-cabundle-cert\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.896371 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.896257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4259\" (UniqueName: \"kubernetes.io/projected/12c31135-f6df-4a38-9306-13579e12ceec-kube-api-access-z4259\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.896371 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.896291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c31135-f6df-4a38-9306-13579e12ceec-kserve-provision-location\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.896371 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:41:08.896341    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 18 20:41:08.896642 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:41:08.896413    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c31135-f6df-4a38-9306-13579e12ceec-proxy-tls podName:12c31135-f6df-4a38-9306-13579e12ceec nodeName:}" failed. No retries permitted until 2026-09-18 20:41:09.396392934 +0000 UTC m=+1238.427319057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12c31135-f6df-4a38-9306-13579e12ceec-proxy-tls") pod "isvc-sklearn-predictor-868898bd6b-5ztvs" (UID: "12c31135-f6df-4a38-9306-13579e12ceec") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 18 20:41:08.896874 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.896736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c31135-f6df-4a38-9306-13579e12ceec-kserve-provision-location\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.896998 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.896977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.897150 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.896977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-cabundle-cert\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:08.905974 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:08.905936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4259\" (UniqueName: \"kubernetes.io/projected/12c31135-f6df-4a38-9306-13579e12ceec-kube-api-access-z4259\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:09.400071 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:09.400035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c31135-f6df-4a38-9306-13579e12ceec-proxy-tls\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:09.402619 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:09.402595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c31135-f6df-4a38-9306-13579e12ceec-proxy-tls\") pod \"isvc-sklearn-predictor-868898bd6b-5ztvs\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:09.605003 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:09.604966    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:09.886070 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:09.886037    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"]
Sep 18 20:41:09.889339 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:41:09.889313    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c31135_f6df_4a38_9306_13579e12ceec.slice/crio-a9950be5e946f32c41501eca2acf93f946177be89dc3b183bd98bb287cc97c79 WatchSource:0}: Error finding container a9950be5e946f32c41501eca2acf93f946177be89dc3b183bd98bb287cc97c79: Status 404 returned error can't find the container with id a9950be5e946f32c41501eca2acf93f946177be89dc3b183bd98bb287cc97c79
Sep 18 20:41:10.511950 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:10.511914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" event={"ID":"12c31135-f6df-4a38-9306-13579e12ceec","Type":"ContainerStarted","Data":"2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68"}
Sep 18 20:41:10.512131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:10.511971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" event={"ID":"12c31135-f6df-4a38-9306-13579e12ceec","Type":"ContainerStarted","Data":"a9950be5e946f32c41501eca2acf93f946177be89dc3b183bd98bb287cc97c79"}
Sep 18 20:41:14.531708 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:14.531665    2566 generic.go:356] "Generic (PLEG): container finished" podID="12c31135-f6df-4a38-9306-13579e12ceec" containerID="2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68" exitCode=0
Sep 18 20:41:14.532198 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:14.531737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" event={"ID":"12c31135-f6df-4a38-9306-13579e12ceec","Type":"ContainerDied","Data":"2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68"}
Sep 18 20:41:15.539133 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:15.539094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" event={"ID":"12c31135-f6df-4a38-9306-13579e12ceec","Type":"ContainerStarted","Data":"bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db"}
Sep 18 20:41:15.539563 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:15.539166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" event={"ID":"12c31135-f6df-4a38-9306-13579e12ceec","Type":"ContainerStarted","Data":"5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b"}
Sep 18 20:41:15.539563 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:15.539396    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:15.559351 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:15.559311    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podStartSLOduration=3.545432266 podStartE2EDuration="7.559296825s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.013864559s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:41:15.558197725 +0000 UTC m=+1244.589123866" watchObservedRunningTime="2026-09-18 20:41:15.559296825 +0000 UTC m=+1244.590222965"
Sep 18 20:41:16.543950 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:16.543917    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:16.545461 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:16.545425    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:41:17.547661 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:17.547617    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:41:22.552854 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:22.552822    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:41:22.553477 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:22.553443    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:41:32.553954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:32.553908    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:41:42.554190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:42.554144    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:41:52.554071 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:41:52.554003    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:42:02.553592 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:02.553547    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:42:12.553427 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:12.553381    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:42:22.554271 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:22.554228    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:42:27.456101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:27.455878    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:42:37.456277 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:37.456069    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:42:47.460446 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:47.460415    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:42:47.949681 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:47.949644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" event={"ID":"273fdfa6-fa4d-4a55-83eb-21955d1b1240","Type":"ContainerStarted","Data":"e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9"}
Sep 18 20:42:47.949846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:47.949708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" event={"ID":"273fdfa6-fa4d-4a55-83eb-21955d1b1240","Type":"ContainerStarted","Data":"db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d"}
Sep 18 20:42:47.949846 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:47.949809    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:42:47.970357 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:47.970249    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" podStartSLOduration=1.500570926 podStartE2EDuration="2m11.970229506s" totalImagesPullingTime="1m50.11521786s" totalInitContainerRuntime="20.35444072s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:42:47.96787217 +0000 UTC m=+1336.998798314" watchObservedRunningTime="2026-09-18 20:42:47.970229506 +0000 UTC m=+1337.001155646"
Sep 18 20:42:48.953809 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:48.953771    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:42:49.126775 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:49.126745    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:42:51.136939 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:51.136907    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:42:53.147579 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:53.147546    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:42:54.963810 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:54.963737    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:42:55.162647 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:55.162619    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:42:57.178580 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:57.178539    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:42:59.189171 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:42:59.189139    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:01.199767 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:01.199740    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:03.210463 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:03.210426    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:05.220578 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:05.220539    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:07.230811 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:07.230780    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:09.242000 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:09.241967    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:11.252212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:11.252181    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:13.262197 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:13.262164    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:15.272941 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:15.272913    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:17.283445 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:17.283411    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:19.293656 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:19.293621    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-868898bd6b-5ztvs_12c31135-f6df-4a38-9306-13579e12ceec/kserve-container/0.log"
Sep 18 20:43:19.447530 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:19.447491    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"]
Sep 18 20:43:19.447738 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:19.447715    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container" containerID="cri-o://5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b" gracePeriod=30
Sep 18 20:43:19.447835 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:19.447784    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kube-rbac-proxy" containerID="cri-o://bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db" gracePeriod=30
Sep 18 20:43:20.076532 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:20.076501    2566 generic.go:356] "Generic (PLEG): container finished" podID="12c31135-f6df-4a38-9306-13579e12ceec" containerID="bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db" exitCode=2
Sep 18 20:43:20.076694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:20.076570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" event={"ID":"12c31135-f6df-4a38-9306-13579e12ceec","Type":"ContainerDied","Data":"bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db"}
Sep 18 20:43:22.548170 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:22.548128    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 18 20:43:24.079720 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.079699    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:43:24.094055 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.094029    2566 generic.go:356] "Generic (PLEG): container finished" podID="12c31135-f6df-4a38-9306-13579e12ceec" containerID="5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b" exitCode=0
Sep 18 20:43:24.094189 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.094079    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"
Sep 18 20:43:24.094189 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.094101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" event={"ID":"12c31135-f6df-4a38-9306-13579e12ceec","Type":"ContainerDied","Data":"5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b"}
Sep 18 20:43:24.094189 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.094155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs" event={"ID":"12c31135-f6df-4a38-9306-13579e12ceec","Type":"ContainerDied","Data":"a9950be5e946f32c41501eca2acf93f946177be89dc3b183bd98bb287cc97c79"}
Sep 18 20:43:24.094189 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.094189    2566 scope.go:117] "RemoveContainer" containerID="bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db"
Sep 18 20:43:24.101868 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.101849    2566 scope.go:117] "RemoveContainer" containerID="5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b"
Sep 18 20:43:24.111469 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.111455    2566 scope.go:117] "RemoveContainer" containerID="2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68"
Sep 18 20:43:24.118675 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.118609    2566 scope.go:117] "RemoveContainer" containerID="bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db"
Sep 18 20:43:24.118951 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:43:24.118931    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db\": container with ID starting with bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db not found: ID does not exist" containerID="bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db"
Sep 18 20:43:24.119005 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.118963    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db"} err="failed to get container status \"bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db\": rpc error: code = NotFound desc = could not find container \"bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db\": container with ID starting with bc8fafb0bdc33e116ba176a1a9cc83f8568a285b75ef761e230ab91bab8f06db not found: ID does not exist"
Sep 18 20:43:24.119005 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.118979    2566 scope.go:117] "RemoveContainer" containerID="5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b"
Sep 18 20:43:24.119240 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:43:24.119223    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b\": container with ID starting with 5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b not found: ID does not exist" containerID="5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b"
Sep 18 20:43:24.119296 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.119251    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b"} err="failed to get container status \"5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b\": rpc error: code = NotFound desc = could not find container \"5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b\": container with ID starting with 5113dae9162915774e2420f7abe1e67b49e48b97764eb6b47b37101b3a3e5e9b not found: ID does not exist"
Sep 18 20:43:24.119296 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.119265    2566 scope.go:117] "RemoveContainer" containerID="2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68"
Sep 18 20:43:24.119554 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:43:24.119535    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68\": container with ID starting with 2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68 not found: ID does not exist" containerID="2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68"
Sep 18 20:43:24.119596 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.119559    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68"} err="failed to get container status \"2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68\": rpc error: code = NotFound desc = could not find container \"2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68\": container with ID starting with 2ea632bd13cece8c02d15408995efd1a06e8aa3503a27830458439e97a7a4e68 not found: ID does not exist"
Sep 18 20:43:24.175766 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.175718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c31135-f6df-4a38-9306-13579e12ceec-kserve-provision-location\") pod \"12c31135-f6df-4a38-9306-13579e12ceec\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") "
Sep 18 20:43:24.175766 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.175750    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-cabundle-cert\") pod \"12c31135-f6df-4a38-9306-13579e12ceec\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") "
Sep 18 20:43:24.175921 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.175785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c31135-f6df-4a38-9306-13579e12ceec-proxy-tls\") pod \"12c31135-f6df-4a38-9306-13579e12ceec\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") "
Sep 18 20:43:24.175921 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.175812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-kube-rbac-proxy-sar-config\") pod \"12c31135-f6df-4a38-9306-13579e12ceec\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") "
Sep 18 20:43:24.175921 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.175831    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4259\" (UniqueName: \"kubernetes.io/projected/12c31135-f6df-4a38-9306-13579e12ceec-kube-api-access-z4259\") pod \"12c31135-f6df-4a38-9306-13579e12ceec\" (UID: \"12c31135-f6df-4a38-9306-13579e12ceec\") "
Sep 18 20:43:24.176120 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.176095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c31135-f6df-4a38-9306-13579e12ceec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12c31135-f6df-4a38-9306-13579e12ceec" (UID: "12c31135-f6df-4a38-9306-13579e12ceec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:43:24.176167 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.176143    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12c31135-f6df-4a38-9306-13579e12ceec" (UID: "12c31135-f6df-4a38-9306-13579e12ceec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:43:24.176167 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.176147    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12c31135-f6df-4a38-9306-13579e12ceec" (UID: "12c31135-f6df-4a38-9306-13579e12ceec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:43:24.177655 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.177634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c31135-f6df-4a38-9306-13579e12ceec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12c31135-f6df-4a38-9306-13579e12ceec" (UID: "12c31135-f6df-4a38-9306-13579e12ceec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:43:24.177789 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.177761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c31135-f6df-4a38-9306-13579e12ceec-kube-api-access-z4259" (OuterVolumeSpecName: "kube-api-access-z4259") pod "12c31135-f6df-4a38-9306-13579e12ceec" (UID: "12c31135-f6df-4a38-9306-13579e12ceec"). InnerVolumeSpecName "kube-api-access-z4259". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:43:24.277267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.277241    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:43:24.277267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.277261    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4259\" (UniqueName: \"kubernetes.io/projected/12c31135-f6df-4a38-9306-13579e12ceec-kube-api-access-z4259\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:43:24.277267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.277271    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c31135-f6df-4a38-9306-13579e12ceec-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:43:24.277444 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.277296    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c31135-f6df-4a38-9306-13579e12ceec-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:43:24.277444 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.277305    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c31135-f6df-4a38-9306-13579e12ceec-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:43:24.415901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.415874    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"]
Sep 18 20:43:24.419627 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:24.419606    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868898bd6b-5ztvs"]
Sep 18 20:43:25.459749 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:43:25.459714    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c31135-f6df-4a38-9306-13579e12ceec" path="/var/lib/kubelet/pods/12c31135-f6df-4a38-9306-13579e12ceec/volumes"
Sep 18 20:44:20.489466 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:20.489432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:44:31.496200 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:31.496114    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:44:32.859390 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:32.859355    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"]
Sep 18 20:44:32.859850 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:32.859703    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="kserve-container" containerID="cri-o://db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d" gracePeriod=30
Sep 18 20:44:32.859850 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:32.859777    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="kube-rbac-proxy" containerID="cri-o://e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9" gracePeriod=30
Sep 18 20:44:33.363765 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:33.363729    2566 generic.go:356] "Generic (PLEG): container finished" podID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerID="e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9" exitCode=2
Sep 18 20:44:33.363980 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:33.363795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" event={"ID":"273fdfa6-fa4d-4a55-83eb-21955d1b1240","Type":"ContainerDied","Data":"e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9"}
Sep 18 20:44:34.292139 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.292120    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:44:34.333770 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.333708    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-cabundle-cert\") pod \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") "
Sep 18 20:44:34.333770 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.333759    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-rbac-proxy-sar-config\") pod \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") "
Sep 18 20:44:34.333950 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.333818    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273fdfa6-fa4d-4a55-83eb-21955d1b1240-proxy-tls\") pod \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") "
Sep 18 20:44:34.333950 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.333853    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsng8\" (UniqueName: \"kubernetes.io/projected/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-api-access-hsng8\") pod \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") "
Sep 18 20:44:34.333950 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.333888    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kserve-provision-location\") pod \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\" (UID: \"273fdfa6-fa4d-4a55-83eb-21955d1b1240\") "
Sep 18 20:44:34.334139 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.334093    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "273fdfa6-fa4d-4a55-83eb-21955d1b1240" (UID: "273fdfa6-fa4d-4a55-83eb-21955d1b1240"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:44:34.334192 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.334158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "273fdfa6-fa4d-4a55-83eb-21955d1b1240" (UID: "273fdfa6-fa4d-4a55-83eb-21955d1b1240"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:44:34.335857 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.335832    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/273fdfa6-fa4d-4a55-83eb-21955d1b1240-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "273fdfa6-fa4d-4a55-83eb-21955d1b1240" (UID: "273fdfa6-fa4d-4a55-83eb-21955d1b1240"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:44:34.335952 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.335875    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-api-access-hsng8" (OuterVolumeSpecName: "kube-api-access-hsng8") pod "273fdfa6-fa4d-4a55-83eb-21955d1b1240" (UID: "273fdfa6-fa4d-4a55-83eb-21955d1b1240"). InnerVolumeSpecName "kube-api-access-hsng8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:44:34.337336 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.337313    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "273fdfa6-fa4d-4a55-83eb-21955d1b1240" (UID: "273fdfa6-fa4d-4a55-83eb-21955d1b1240"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:34.368932 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.368908    2566 generic.go:356] "Generic (PLEG): container finished" podID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerID="db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d" exitCode=0
Sep 18 20:44:34.369050 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.368943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" event={"ID":"273fdfa6-fa4d-4a55-83eb-21955d1b1240","Type":"ContainerDied","Data":"db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d"}
Sep 18 20:44:34.369050 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.368962    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"
Sep 18 20:44:34.369050 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.368989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5" event={"ID":"273fdfa6-fa4d-4a55-83eb-21955d1b1240","Type":"ContainerDied","Data":"95c6ca44e47073e388eb4ddcf04070e653c251da49aaf9c3758f59fcfa5d98cc"}
Sep 18 20:44:34.369050 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.369026    2566 scope.go:117] "RemoveContainer" containerID="e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9"
Sep 18 20:44:34.377639 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.377623    2566 scope.go:117] "RemoveContainer" containerID="db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d"
Sep 18 20:44:34.384752 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.384737    2566 scope.go:117] "RemoveContainer" containerID="c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f"
Sep 18 20:44:34.391784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.391767    2566 scope.go:117] "RemoveContainer" containerID="e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9"
Sep 18 20:44:34.391878 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.391858    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"]
Sep 18 20:44:34.392088 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:44:34.392069    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9\": container with ID starting with e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9 not found: ID does not exist" containerID="e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9"
Sep 18 20:44:34.392149 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.392094    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9"} err="failed to get container status \"e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9\": rpc error: code = NotFound desc = could not find container \"e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9\": container with ID starting with e20a1bef616c3ae2d0b0f764dfca4f4e6795a99bdb2b66b4b7da0f3ad1b5f2b9 not found: ID does not exist"
Sep 18 20:44:34.392149 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.392109    2566 scope.go:117] "RemoveContainer" containerID="db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d"
Sep 18 20:44:34.392354 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:44:34.392341    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d\": container with ID starting with db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d not found: ID does not exist" containerID="db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d"
Sep 18 20:44:34.392397 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.392356    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d"} err="failed to get container status \"db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d\": rpc error: code = NotFound desc = could not find container \"db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d\": container with ID starting with db24677b4eeac064bdce1dfde1b5ef18f748d82eddbd1a7b9480233c7d24294d not found: ID does not exist"
Sep 18 20:44:34.392397 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.392370    2566 scope.go:117] "RemoveContainer" containerID="c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f"
Sep 18 20:44:34.392588 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:44:34.392574    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f\": container with ID starting with c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f not found: ID does not exist" containerID="c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f"
Sep 18 20:44:34.392630 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.392590    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f"} err="failed to get container status \"c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f\": rpc error: code = NotFound desc = could not find container \"c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f\": container with ID starting with c4f6f06ec5ddf1c1b88d6e171bb32e923fe61f9ead18606797beaaaab2767d6f not found: ID does not exist"
Sep 18 20:44:34.397325 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.397307    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tcdw5"]
Sep 18 20:44:34.435313 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.435289    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/273fdfa6-fa4d-4a55-83eb-21955d1b1240-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:44:34.435313 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.435311    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsng8\" (UniqueName: \"kubernetes.io/projected/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-api-access-hsng8\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:44:34.435437 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.435321    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:44:34.435437 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.435330    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:44:34.435437 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:34.435338    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/273fdfa6-fa4d-4a55-83eb-21955d1b1240-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:44:35.459899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:35.459868    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" path="/var/lib/kubelet/pods/273fdfa6-fa4d-4a55-83eb-21955d1b1240/volumes"
Sep 18 20:44:38.053763 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.053729    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"]
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054083    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="kube-rbac-proxy"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054096    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="kube-rbac-proxy"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054111    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054117    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054125    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kube-rbac-proxy"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054130    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kube-rbac-proxy"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054138    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="storage-initializer"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054144    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="storage-initializer"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054154    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="kserve-container"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054159    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="kserve-container"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054165    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="storage-initializer"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054170    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="storage-initializer"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054214    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kube-rbac-proxy"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054221    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c31135-f6df-4a38-9306-13579e12ceec" containerName="kserve-container"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054227    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="kserve-container"
Sep 18 20:44:38.054319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.054234    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="273fdfa6-fa4d-4a55-83eb-21955d1b1240" containerName="kube-rbac-proxy"
Sep 18 20:44:38.057107 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.057088    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.060029 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.059998    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:44:38.060143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.060036    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:44:38.060232 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.060218    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:44:38.060287 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.060255    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:44:38.060287 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.060221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 18 20:44:38.061219 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.061204    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 18 20:44:38.061305 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.061238    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:44:38.067815 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.067795    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"]
Sep 18 20:44:38.162418 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.162385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshn9\" (UniqueName: \"kubernetes.io/projected/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-api-access-rshn9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.162593 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.162427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.162593 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.162475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.162593 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.162500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/669813ea-5c17-4a1a-bf0a-6608c39e2876-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.162593 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.162521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669813ea-5c17-4a1a-bf0a-6608c39e2876-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.263382 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.263344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/669813ea-5c17-4a1a-bf0a-6608c39e2876-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.263382 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.263384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669813ea-5c17-4a1a-bf0a-6608c39e2876-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.263632 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.263415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshn9\" (UniqueName: \"kubernetes.io/projected/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-api-access-rshn9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.263632 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.263449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.263632 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.263487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.263632 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:44:38.263524    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 18 20:44:38.263632 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:44:38.263630    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669813ea-5c17-4a1a-bf0a-6608c39e2876-proxy-tls podName:669813ea-5c17-4a1a-bf0a-6608c39e2876 nodeName:}" failed. No retries permitted until 2026-09-18 20:44:38.76360678 +0000 UTC m=+1447.794532912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/669813ea-5c17-4a1a-bf0a-6608c39e2876-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" (UID: "669813ea-5c17-4a1a-bf0a-6608c39e2876") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 18 20:44:38.263945 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.263812    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669813ea-5c17-4a1a-bf0a-6608c39e2876-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.264127 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.264107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.264254 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.264236    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.274397 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.274378    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshn9\" (UniqueName: \"kubernetes.io/projected/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-api-access-rshn9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.767919 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.767886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/669813ea-5c17-4a1a-bf0a-6608c39e2876-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.770196 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.770173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/669813ea-5c17-4a1a-bf0a-6608c39e2876-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:38.967668 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:38.967628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:44:39.239868 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:39.239779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"]
Sep 18 20:44:39.241876 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:44:39.241838    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669813ea_5c17_4a1a_bf0a_6608c39e2876.slice/crio-5793379b055faac6fccee07e9da34de1481a2f86e2d640176cd881929addaaa3 WatchSource:0}: Error finding container 5793379b055faac6fccee07e9da34de1481a2f86e2d640176cd881929addaaa3: Status 404 returned error can't find the container with id 5793379b055faac6fccee07e9da34de1481a2f86e2d640176cd881929addaaa3
Sep 18 20:44:39.392745 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:39.392711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" event={"ID":"669813ea-5c17-4a1a-bf0a-6608c39e2876","Type":"ContainerStarted","Data":"4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01"}
Sep 18 20:44:39.392914 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:44:39.392782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" event={"ID":"669813ea-5c17-4a1a-bf0a-6608c39e2876","Type":"ContainerStarted","Data":"5793379b055faac6fccee07e9da34de1481a2f86e2d640176cd881929addaaa3"}
Sep 18 20:45:09.505934 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:09.505901    2566 generic.go:356] "Generic (PLEG): container finished" podID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerID="4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01" exitCode=0
Sep 18 20:45:09.506350 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:09.505972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" event={"ID":"669813ea-5c17-4a1a-bf0a-6608c39e2876","Type":"ContainerDied","Data":"4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01"}
Sep 18 20:45:10.512169 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:10.512137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" event={"ID":"669813ea-5c17-4a1a-bf0a-6608c39e2876","Type":"ContainerStarted","Data":"54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54"}
Sep 18 20:45:10.512558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:10.512196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" event={"ID":"669813ea-5c17-4a1a-bf0a-6608c39e2876","Type":"ContainerStarted","Data":"99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13"}
Sep 18 20:45:10.512558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:10.512388    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:45:10.530868 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:10.530817    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" podStartSLOduration=3.111142092 podStartE2EDuration="32.530802132s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.41966004s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:45:10.528849468 +0000 UTC m=+1479.559775608" watchObservedRunningTime="2026-09-18 20:45:10.530802132 +0000 UTC m=+1479.561728272"
Sep 18 20:45:11.516653 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:11.516624    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:45:17.525437 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:17.525408    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:45:31.444476 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:31.444448    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:45:31.448334 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:45:31.448314    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:46:29.946510 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:29.946473    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"]
Sep 18 20:46:29.949841 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:29.949824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:29.952496 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:29.952464    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 18 20:46:29.952653 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:29.952498    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:46:29.961262 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:29.961237    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"]
Sep 18 20:46:30.018378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.018350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.018526 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.018391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.018526 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.018432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090b75aa-a74b-4f63-9863-0b7fcb196f1d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.018526 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.018464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2smx\" (UniqueName: \"kubernetes.io/projected/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-api-access-g2smx\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.018526 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.018495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.119648 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.119611    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.119648 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.119652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090b75aa-a74b-4f63-9863-0b7fcb196f1d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.119895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.119679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2smx\" (UniqueName: \"kubernetes.io/projected/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-api-access-g2smx\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.119895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.119713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.119895 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.119769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.119895 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:46:30.119770    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 18 20:46:30.119895 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:46:30.119877    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/090b75aa-a74b-4f63-9863-0b7fcb196f1d-proxy-tls podName:090b75aa-a74b-4f63-9863-0b7fcb196f1d nodeName:}" failed. No retries permitted until 2026-09-18 20:46:30.619856794 +0000 UTC m=+1559.650782914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/090b75aa-a74b-4f63-9863-0b7fcb196f1d-proxy-tls") pod "isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" (UID: "090b75aa-a74b-4f63-9863-0b7fcb196f1d") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 18 20:46:30.120210 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.120184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.120405 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.120383    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.120505 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.120488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.129004 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.128981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2smx\" (UniqueName: \"kubernetes.io/projected/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-api-access-g2smx\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.623057 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.623000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090b75aa-a74b-4f63-9863-0b7fcb196f1d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.625463 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.625436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090b75aa-a74b-4f63-9863-0b7fcb196f1d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:30.859783 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:30.859751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:31.130433 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:31.130404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"]
Sep 18 20:46:31.132580 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:46:31.132546    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090b75aa_a74b_4f63_9863_0b7fcb196f1d.slice/crio-c86bf860114d13991a2efd597e900ecfb093494f3e45081753771e6ccb5b30e6 WatchSource:0}: Error finding container c86bf860114d13991a2efd597e900ecfb093494f3e45081753771e6ccb5b30e6: Status 404 returned error can't find the container with id c86bf860114d13991a2efd597e900ecfb093494f3e45081753771e6ccb5b30e6
Sep 18 20:46:31.829877 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:31.829840    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" event={"ID":"090b75aa-a74b-4f63-9863-0b7fcb196f1d","Type":"ContainerStarted","Data":"baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb"}
Sep 18 20:46:31.832003 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:31.829902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" event={"ID":"090b75aa-a74b-4f63-9863-0b7fcb196f1d","Type":"ContainerStarted","Data":"c86bf860114d13991a2efd597e900ecfb093494f3e45081753771e6ccb5b30e6"}
Sep 18 20:46:36.851238 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:36.851202    2566 generic.go:356] "Generic (PLEG): container finished" podID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerID="baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb" exitCode=0
Sep 18 20:46:36.851668 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:36.851272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" event={"ID":"090b75aa-a74b-4f63-9863-0b7fcb196f1d","Type":"ContainerDied","Data":"baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb"}
Sep 18 20:46:37.857604 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:37.857569    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" event={"ID":"090b75aa-a74b-4f63-9863-0b7fcb196f1d","Type":"ContainerStarted","Data":"8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40"}
Sep 18 20:46:37.857969 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:37.857631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" event={"ID":"090b75aa-a74b-4f63-9863-0b7fcb196f1d","Type":"ContainerStarted","Data":"d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900"}
Sep 18 20:46:37.857969 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:37.857915    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:37.858072 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:37.858057    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:37.859629 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:37.859598    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:46:37.877747 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:37.877705    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" podStartSLOduration=3.704212109 podStartE2EDuration="8.87769215s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.173480041s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:46:37.87580068 +0000 UTC m=+1566.906726845" watchObservedRunningTime="2026-09-18 20:46:37.87769215 +0000 UTC m=+1566.908618289"
Sep 18 20:46:38.861729 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:38.861686    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:46:43.865911 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:43.865881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:43.866421 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:43.866392    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 18 20:46:48.487261 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:48.487236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:46:53.867484 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:53.867449    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:46:59.497190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:46:59.497144    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:47:02.100791 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:02.100760    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"]
Sep 18 20:47:02.101213 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:02.100994    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kserve-container" containerID="cri-o://d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900" gracePeriod=30
Sep 18 20:47:02.101279 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:02.101195    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kube-rbac-proxy" containerID="cri-o://8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40" gracePeriod=30
Sep 18 20:47:02.965976 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:02.965929    2566 generic.go:356] "Generic (PLEG): container finished" podID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerID="8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40" exitCode=2
Sep 18 20:47:02.966163 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:02.966000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" event={"ID":"090b75aa-a74b-4f63-9863-0b7fcb196f1d","Type":"ContainerDied","Data":"8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40"}
Sep 18 20:47:03.345571 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.345544    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:47:03.493967 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.493890    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090b75aa-a74b-4f63-9863-0b7fcb196f1d-proxy-tls\") pod \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") "
Sep 18 20:47:03.493967 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.493933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-rbac-proxy-sar-config\") pod \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") "
Sep 18 20:47:03.493967 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.493958    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kserve-provision-location\") pod \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") "
Sep 18 20:47:03.494273 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.494004    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-cabundle-cert\") pod \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") "
Sep 18 20:47:03.494273 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.494077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2smx\" (UniqueName: \"kubernetes.io/projected/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-api-access-g2smx\") pod \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\" (UID: \"090b75aa-a74b-4f63-9863-0b7fcb196f1d\") "
Sep 18 20:47:03.494380 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.494354    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "090b75aa-a74b-4f63-9863-0b7fcb196f1d" (UID: "090b75aa-a74b-4f63-9863-0b7fcb196f1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:47:03.494492 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.494468    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "090b75aa-a74b-4f63-9863-0b7fcb196f1d" (UID: "090b75aa-a74b-4f63-9863-0b7fcb196f1d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:47:03.496169 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.496144    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-api-access-g2smx" (OuterVolumeSpecName: "kube-api-access-g2smx") pod "090b75aa-a74b-4f63-9863-0b7fcb196f1d" (UID: "090b75aa-a74b-4f63-9863-0b7fcb196f1d"). InnerVolumeSpecName "kube-api-access-g2smx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:47:03.496169 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.496152    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090b75aa-a74b-4f63-9863-0b7fcb196f1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "090b75aa-a74b-4f63-9863-0b7fcb196f1d" (UID: "090b75aa-a74b-4f63-9863-0b7fcb196f1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:47:03.534615 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.534581    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "090b75aa-a74b-4f63-9863-0b7fcb196f1d" (UID: "090b75aa-a74b-4f63-9863-0b7fcb196f1d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:03.595517 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.595492    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2smx\" (UniqueName: \"kubernetes.io/projected/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-api-access-g2smx\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:03.595517 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.595515    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090b75aa-a74b-4f63-9863-0b7fcb196f1d-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:03.595713 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.595527    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:03.595713 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.595536    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090b75aa-a74b-4f63-9863-0b7fcb196f1d-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:03.595713 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.595544    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090b75aa-a74b-4f63-9863-0b7fcb196f1d-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:03.972916 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.972881    2566 generic.go:356] "Generic (PLEG): container finished" podID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerID="d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900" exitCode=0
Sep 18 20:47:03.973090 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.972941    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"
Sep 18 20:47:03.973090 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.972977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" event={"ID":"090b75aa-a74b-4f63-9863-0b7fcb196f1d","Type":"ContainerDied","Data":"d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900"}
Sep 18 20:47:03.973090 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.973066    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9" event={"ID":"090b75aa-a74b-4f63-9863-0b7fcb196f1d","Type":"ContainerDied","Data":"c86bf860114d13991a2efd597e900ecfb093494f3e45081753771e6ccb5b30e6"}
Sep 18 20:47:03.973208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.973109    2566 scope.go:117] "RemoveContainer" containerID="8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40"
Sep 18 20:47:03.984027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.983983    2566 scope.go:117] "RemoveContainer" containerID="d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900"
Sep 18 20:47:03.992323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.992306    2566 scope.go:117] "RemoveContainer" containerID="baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb"
Sep 18 20:47:03.999533 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:03.999506    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"]
Sep 18 20:47:04.001101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.001057    2566 scope.go:117] "RemoveContainer" containerID="8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40"
Sep 18 20:47:04.001374 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:47:04.001357    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40\": container with ID starting with 8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40 not found: ID does not exist" containerID="8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40"
Sep 18 20:47:04.001445 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.001382    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40"} err="failed to get container status \"8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40\": rpc error: code = NotFound desc = could not find container \"8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40\": container with ID starting with 8a8d721fb3b93dc79cb676c39918c44487b67ec551af4ff1160a020ceb7d9d40 not found: ID does not exist"
Sep 18 20:47:04.001445 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.001398    2566 scope.go:117] "RemoveContainer" containerID="d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900"
Sep 18 20:47:04.001660 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:47:04.001640    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900\": container with ID starting with d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900 not found: ID does not exist" containerID="d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900"
Sep 18 20:47:04.001725 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.001664    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900"} err="failed to get container status \"d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900\": rpc error: code = NotFound desc = could not find container \"d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900\": container with ID starting with d613c2bc86f2a8c91c05167d5fd23842fbeb38a6b038fa8997f01b6e3b1ac900 not found: ID does not exist"
Sep 18 20:47:04.001725 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.001679    2566 scope.go:117] "RemoveContainer" containerID="baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb"
Sep 18 20:47:04.001953 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:47:04.001930    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb\": container with ID starting with baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb not found: ID does not exist" containerID="baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb"
Sep 18 20:47:04.002034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.001958    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb"} err="failed to get container status \"baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb\": rpc error: code = NotFound desc = could not find container \"baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb\": container with ID starting with baa6675ccd50ab894e77024f844f65e0e54b1b411f91b814d4a6b88279171cfb not found: ID does not exist"
Sep 18 20:47:04.002478 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.002458    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6ff5866d8b-jzws9"]
Sep 18 20:47:04.540197 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.540163    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"]
Sep 18 20:47:04.540671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.540449    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="kserve-container" containerID="cri-o://99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13" gracePeriod=30
Sep 18 20:47:04.540671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.540615    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="kube-rbac-proxy" containerID="cri-o://54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54" gracePeriod=30
Sep 18 20:47:04.979732 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.979645    2566 generic.go:356] "Generic (PLEG): container finished" podID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerID="54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54" exitCode=2
Sep 18 20:47:04.979885 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:04.979721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" event={"ID":"669813ea-5c17-4a1a-bf0a-6608c39e2876","Type":"ContainerDied","Data":"54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54"}
Sep 18 20:47:05.459420 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:05.459377    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" path="/var/lib/kubelet/pods/090b75aa-a74b-4f63-9863-0b7fcb196f1d/volumes"
Sep 18 20:47:05.983851 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:05.983827    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:47:05.985668 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:05.985646    2566 generic.go:356] "Generic (PLEG): container finished" podID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerID="99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13" exitCode=0
Sep 18 20:47:05.985759 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:05.985717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" event={"ID":"669813ea-5c17-4a1a-bf0a-6608c39e2876","Type":"ContainerDied","Data":"99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13"}
Sep 18 20:47:05.985813 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:05.985784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9" event={"ID":"669813ea-5c17-4a1a-bf0a-6608c39e2876","Type":"ContainerDied","Data":"5793379b055faac6fccee07e9da34de1481a2f86e2d640176cd881929addaaa3"}
Sep 18 20:47:05.985849 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:05.985812    2566 scope.go:117] "RemoveContainer" containerID="54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54"
Sep 18 20:47:05.993320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:05.993301    2566 scope.go:117] "RemoveContainer" containerID="99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13"
Sep 18 20:47:06.000205 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.000190    2566 scope.go:117] "RemoveContainer" containerID="4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01"
Sep 18 20:47:06.007911 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.007887    2566 scope.go:117] "RemoveContainer" containerID="54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54"
Sep 18 20:47:06.008244 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:47:06.008219    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54\": container with ID starting with 54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54 not found: ID does not exist" containerID="54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54"
Sep 18 20:47:06.008340 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.008253    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54"} err="failed to get container status \"54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54\": rpc error: code = NotFound desc = could not find container \"54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54\": container with ID starting with 54cece30b804cf224aa221a71fe4debbab61b25f05978931c179d27d24066e54 not found: ID does not exist"
Sep 18 20:47:06.008340 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.008277    2566 scope.go:117] "RemoveContainer" containerID="99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13"
Sep 18 20:47:06.008555 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:47:06.008531    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13\": container with ID starting with 99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13 not found: ID does not exist" containerID="99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13"
Sep 18 20:47:06.008659 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.008559    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13"} err="failed to get container status \"99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13\": rpc error: code = NotFound desc = could not find container \"99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13\": container with ID starting with 99c8bc3c3bdbd0ed8fe48be9c26fc51122613644d7b5ce728237f4ed3f394c13 not found: ID does not exist"
Sep 18 20:47:06.008659 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.008578    2566 scope.go:117] "RemoveContainer" containerID="4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01"
Sep 18 20:47:06.008805 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:47:06.008787    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01\": container with ID starting with 4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01 not found: ID does not exist" containerID="4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01"
Sep 18 20:47:06.008872 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.008816    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01"} err="failed to get container status \"4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01\": rpc error: code = NotFound desc = could not find container \"4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01\": container with ID starting with 4e346c65ba70129f4e195e71dc3cb0d1126f0ee6bb210f4fe11aec22558b5a01 not found: ID does not exist"
Sep 18 20:47:06.116051 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.115975    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-rbac-proxy-sar-config\") pod \"669813ea-5c17-4a1a-bf0a-6608c39e2876\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") "
Sep 18 20:47:06.116051 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.116008    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669813ea-5c17-4a1a-bf0a-6608c39e2876-kserve-provision-location\") pod \"669813ea-5c17-4a1a-bf0a-6608c39e2876\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") "
Sep 18 20:47:06.116205 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.116141    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-cabundle-cert\") pod \"669813ea-5c17-4a1a-bf0a-6608c39e2876\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") "
Sep 18 20:47:06.116205 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.116174    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/669813ea-5c17-4a1a-bf0a-6608c39e2876-proxy-tls\") pod \"669813ea-5c17-4a1a-bf0a-6608c39e2876\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") "
Sep 18 20:47:06.116302 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.116213    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rshn9\" (UniqueName: \"kubernetes.io/projected/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-api-access-rshn9\") pod \"669813ea-5c17-4a1a-bf0a-6608c39e2876\" (UID: \"669813ea-5c17-4a1a-bf0a-6608c39e2876\") "
Sep 18 20:47:06.116421 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.116397    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "669813ea-5c17-4a1a-bf0a-6608c39e2876" (UID: "669813ea-5c17-4a1a-bf0a-6608c39e2876"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:47:06.116491 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.116467    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "669813ea-5c17-4a1a-bf0a-6608c39e2876" (UID: "669813ea-5c17-4a1a-bf0a-6608c39e2876"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:47:06.118312 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.118287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-api-access-rshn9" (OuterVolumeSpecName: "kube-api-access-rshn9") pod "669813ea-5c17-4a1a-bf0a-6608c39e2876" (UID: "669813ea-5c17-4a1a-bf0a-6608c39e2876"). InnerVolumeSpecName "kube-api-access-rshn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:47:06.118402 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.118287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669813ea-5c17-4a1a-bf0a-6608c39e2876-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "669813ea-5c17-4a1a-bf0a-6608c39e2876" (UID: "669813ea-5c17-4a1a-bf0a-6608c39e2876"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:47:06.120152 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.120133    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669813ea-5c17-4a1a-bf0a-6608c39e2876-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "669813ea-5c17-4a1a-bf0a-6608c39e2876" (UID: "669813ea-5c17-4a1a-bf0a-6608c39e2876"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:06.217292 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.217265    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:06.217292 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.217289    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669813ea-5c17-4a1a-bf0a-6608c39e2876-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:06.217438 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.217302    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669813ea-5c17-4a1a-bf0a-6608c39e2876-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:06.217438 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.217314    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/669813ea-5c17-4a1a-bf0a-6608c39e2876-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:06.217438 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.217324    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshn9\" (UniqueName: \"kubernetes.io/projected/669813ea-5c17-4a1a-bf0a-6608c39e2876-kube-api-access-rshn9\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:06.990567 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:06.990539    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"
Sep 18 20:47:07.011977 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:07.011949    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"]
Sep 18 20:47:07.017838 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:07.017813    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-c6sx9"]
Sep 18 20:47:07.465462 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:07.465421    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" path="/var/lib/kubelet/pods/669813ea-5c17-4a1a-bf0a-6608c39e2876/volumes"
Sep 18 20:47:09.688808 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.688776    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"]
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689061    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kube-rbac-proxy"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689072    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kube-rbac-proxy"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689081    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="storage-initializer"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689088    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="storage-initializer"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689098    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="storage-initializer"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689103    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="storage-initializer"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689113    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kserve-container"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689118    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kserve-container"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689126    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="kube-rbac-proxy"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689131    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="kube-rbac-proxy"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689140    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="kserve-container"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689146    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="kserve-container"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689194    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kserve-container"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689204    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="kube-rbac-proxy"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689210    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="669813ea-5c17-4a1a-bf0a-6608c39e2876" containerName="kserve-container"
Sep 18 20:47:09.689235 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.689215    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="090b75aa-a74b-4f63-9863-0b7fcb196f1d" containerName="kube-rbac-proxy"
Sep 18 20:47:09.693994 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.693975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.696857 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.696815    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 18 20:47:09.696857 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.696822    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:47:09.696857 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.696854    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:47:09.697059 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.696875    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:47:09.698005 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.697988    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 18 20:47:09.698079 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.698065    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:47:09.698227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.698214    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:47:09.702142 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.702119    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"]
Sep 18 20:47:09.847151 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.847118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.847318 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.847168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf67ce2b-2a46-47f6-9da9-14dc86c788af-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.847318 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.847186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.847318 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.847251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjls\" (UniqueName: \"kubernetes.io/projected/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-api-access-2wjls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.847318 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.847299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.947899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.947815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf67ce2b-2a46-47f6-9da9-14dc86c788af-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.947899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.947855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.947899 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.947891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjls\" (UniqueName: \"kubernetes.io/projected/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-api-access-2wjls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.948199 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.947938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.948199 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.947982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.948387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.948367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.948517 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.948498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.948564 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.948539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.950318 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.950297    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf67ce2b-2a46-47f6-9da9-14dc86c788af-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:09.956542 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:09.956521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjls\" (UniqueName: \"kubernetes.io/projected/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-api-access-2wjls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:10.004869 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:10.004840    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:10.275573 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:10.275548    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"]
Sep 18 20:47:10.277387 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:47:10.277362    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf67ce2b_2a46_47f6_9da9_14dc86c788af.slice/crio-dcca7c39527164e14fac29dcc6120a7c8fa52b6f0a04072e43a5ce1c09cdcc65 WatchSource:0}: Error finding container dcca7c39527164e14fac29dcc6120a7c8fa52b6f0a04072e43a5ce1c09cdcc65: Status 404 returned error can't find the container with id dcca7c39527164e14fac29dcc6120a7c8fa52b6f0a04072e43a5ce1c09cdcc65
Sep 18 20:47:11.007500 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:11.007463    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" event={"ID":"cf67ce2b-2a46-47f6-9da9-14dc86c788af","Type":"ContainerStarted","Data":"3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9"}
Sep 18 20:47:11.007850 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:11.007522    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" event={"ID":"cf67ce2b-2a46-47f6-9da9-14dc86c788af","Type":"ContainerStarted","Data":"dcca7c39527164e14fac29dcc6120a7c8fa52b6f0a04072e43a5ce1c09cdcc65"}
Sep 18 20:47:20.041874 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:20.041837    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerID="3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9" exitCode=0
Sep 18 20:47:20.042419 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:20.041912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" event={"ID":"cf67ce2b-2a46-47f6-9da9-14dc86c788af","Type":"ContainerDied","Data":"3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9"}
Sep 18 20:47:21.048741 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:21.048707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" event={"ID":"cf67ce2b-2a46-47f6-9da9-14dc86c788af","Type":"ContainerStarted","Data":"843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57"}
Sep 18 20:47:21.049228 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:21.048773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" event={"ID":"cf67ce2b-2a46-47f6-9da9-14dc86c788af","Type":"ContainerStarted","Data":"4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd"}
Sep 18 20:47:21.049228 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:21.048985    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:21.071600 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:21.071546    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" podStartSLOduration=3.305241897 podStartE2EDuration="12.071527681s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.766285784s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:47:21.068997343 +0000 UTC m=+1610.099923507" watchObservedRunningTime="2026-09-18 20:47:21.071527681 +0000 UTC m=+1610.102453822"
Sep 18 20:47:22.052431 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:22.052400    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:22.053768 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:22.053742    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:47:23.057185 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:23.057142    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:47:28.062270 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:28.062195    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:28.062712 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:28.062687    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 18 20:47:31.530592 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:31.530557    2566 scope.go:117] "RemoveContainer" containerID="dc5a8389f1dff6b00d7680dd3671bd8ad65d6a50627ac56ead75419cbc8b6c17"
Sep 18 20:47:38.064158 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:38.064132    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:52.016994 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:52.016962    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"]
Sep 18 20:47:52.017495 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:52.017218    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kserve-container" containerID="cri-o://4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd" gracePeriod=30
Sep 18 20:47:52.017495 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:52.017245    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kube-rbac-proxy" containerID="cri-o://843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57" gracePeriod=30
Sep 18 20:47:52.181997 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:52.181967    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerID="843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57" exitCode=2
Sep 18 20:47:52.182129 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:52.182040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" event={"ID":"cf67ce2b-2a46-47f6-9da9-14dc86c788af","Type":"ContainerDied","Data":"843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57"}
Sep 18 20:47:53.058211 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.058158    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 18 20:47:53.646963 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.646941    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:53.802790 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.802762    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wjls\" (UniqueName: \"kubernetes.io/projected/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-api-access-2wjls\") pod \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") "
Sep 18 20:47:53.802972 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.802820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kserve-provision-location\") pod \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") "
Sep 18 20:47:53.802972 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.802836    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf67ce2b-2a46-47f6-9da9-14dc86c788af-proxy-tls\") pod \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") "
Sep 18 20:47:53.802972 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.802855    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-rbac-proxy-sar-config\") pod \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") "
Sep 18 20:47:53.802972 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.802873    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-cabundle-cert\") pod \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\" (UID: \"cf67ce2b-2a46-47f6-9da9-14dc86c788af\") "
Sep 18 20:47:53.803281 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.803257    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf67ce2b-2a46-47f6-9da9-14dc86c788af" (UID: "cf67ce2b-2a46-47f6-9da9-14dc86c788af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:47:53.803319 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.803266    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf67ce2b-2a46-47f6-9da9-14dc86c788af" (UID: "cf67ce2b-2a46-47f6-9da9-14dc86c788af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:47:53.803726 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.803706    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf67ce2b-2a46-47f6-9da9-14dc86c788af" (UID: "cf67ce2b-2a46-47f6-9da9-14dc86c788af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:53.804961 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.804942    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-api-access-2wjls" (OuterVolumeSpecName: "kube-api-access-2wjls") pod "cf67ce2b-2a46-47f6-9da9-14dc86c788af" (UID: "cf67ce2b-2a46-47f6-9da9-14dc86c788af"). InnerVolumeSpecName "kube-api-access-2wjls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:47:53.805048 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.804984    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf67ce2b-2a46-47f6-9da9-14dc86c788af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf67ce2b-2a46-47f6-9da9-14dc86c788af" (UID: "cf67ce2b-2a46-47f6-9da9-14dc86c788af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:47:53.903891 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.903870    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:53.903891 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.903889    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf67ce2b-2a46-47f6-9da9-14dc86c788af-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:53.904049 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.903900    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:53.904049 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.903909    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf67ce2b-2a46-47f6-9da9-14dc86c788af-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:53.904049 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:53.903918    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wjls\" (UniqueName: \"kubernetes.io/projected/cf67ce2b-2a46-47f6-9da9-14dc86c788af-kube-api-access-2wjls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:47:54.193154 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.193058    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerID="4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd" exitCode=0
Sep 18 20:47:54.193154 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.193108    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"
Sep 18 20:47:54.193154 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.193134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" event={"ID":"cf67ce2b-2a46-47f6-9da9-14dc86c788af","Type":"ContainerDied","Data":"4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd"}
Sep 18 20:47:54.193662 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.193195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc" event={"ID":"cf67ce2b-2a46-47f6-9da9-14dc86c788af","Type":"ContainerDied","Data":"dcca7c39527164e14fac29dcc6120a7c8fa52b6f0a04072e43a5ce1c09cdcc65"}
Sep 18 20:47:54.193662 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.193237    2566 scope.go:117] "RemoveContainer" containerID="843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57"
Sep 18 20:47:54.201350 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.201211    2566 scope.go:117] "RemoveContainer" containerID="4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd"
Sep 18 20:47:54.208668 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.208651    2566 scope.go:117] "RemoveContainer" containerID="3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9"
Sep 18 20:47:54.215311 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.215294    2566 scope.go:117] "RemoveContainer" containerID="843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57"
Sep 18 20:47:54.215576 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:47:54.215560    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57\": container with ID starting with 843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57 not found: ID does not exist" containerID="843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57"
Sep 18 20:47:54.215629 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.215583    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57"} err="failed to get container status \"843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57\": rpc error: code = NotFound desc = could not find container \"843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57\": container with ID starting with 843d50e117570a749b5ab8f0dc0c490b100e1fe58db22258d71f12541b5dea57 not found: ID does not exist"
Sep 18 20:47:54.215629 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.215599    2566 scope.go:117] "RemoveContainer" containerID="4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd"
Sep 18 20:47:54.215849 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:47:54.215830    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd\": container with ID starting with 4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd not found: ID does not exist" containerID="4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd"
Sep 18 20:47:54.215887 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.215855    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd"} err="failed to get container status \"4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd\": rpc error: code = NotFound desc = could not find container \"4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd\": container with ID starting with 4a79f6af2023c5b3d4e300983d84b802a31565c1203e8512105a14910c665ebd not found: ID does not exist"
Sep 18 20:47:54.215887 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.215872    2566 scope.go:117] "RemoveContainer" containerID="3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9"
Sep 18 20:47:54.216207 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:47:54.216182    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9\": container with ID starting with 3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9 not found: ID does not exist" containerID="3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9"
Sep 18 20:47:54.216317 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.216215    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9"} err="failed to get container status \"3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9\": rpc error: code = NotFound desc = could not find container \"3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9\": container with ID starting with 3fdb45f3f0a596634917eac7d255ff47053e601c7998af6524b34a4a4eea4ce9 not found: ID does not exist"
Sep 18 20:47:54.218536 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.218514    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"]
Sep 18 20:47:54.223927 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:54.223907    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tncnc"]
Sep 18 20:47:55.459892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:55.459855    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" path="/var/lib/kubelet/pods/cf67ce2b-2a46-47f6-9da9-14dc86c788af/volumes"
Sep 18 20:47:57.214824 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.214796    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"]
Sep 18 20:47:57.215190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.215089    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="storage-initializer"
Sep 18 20:47:57.215190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.215103    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="storage-initializer"
Sep 18 20:47:57.215190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.215119    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kserve-container"
Sep 18 20:47:57.215190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.215124    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kserve-container"
Sep 18 20:47:57.215190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.215130    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kube-rbac-proxy"
Sep 18 20:47:57.215190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.215135    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kube-rbac-proxy"
Sep 18 20:47:57.215190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.215177    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kserve-container"
Sep 18 20:47:57.215190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.215187    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf67ce2b-2a46-47f6-9da9-14dc86c788af" containerName="kube-rbac-proxy"
Sep 18 20:47:57.219949 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.219931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.223104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.222903    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 18 20:47:57.223104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.222908    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 18 20:47:57.223104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.222933    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:47:57.223104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.222989    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:47:57.223104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.223003    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:47:57.223104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.222911    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:47:57.223104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.223074    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:47:57.228983 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.228963    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"]
Sep 18 20:47:57.331623 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.331582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.331623 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.331626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.331819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.331654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e37e222-5226-4118-ad77-048d2ca4db00-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.331819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.331751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e37e222-5226-4118-ad77-048d2ca4db00-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.331819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.331799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzz28\" (UniqueName: \"kubernetes.io/projected/1e37e222-5226-4118-ad77-048d2ca4db00-kube-api-access-tzz28\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.433034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.432977    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e37e222-5226-4118-ad77-048d2ca4db00-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.433193 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.433119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzz28\" (UniqueName: \"kubernetes.io/projected/1e37e222-5226-4118-ad77-048d2ca4db00-kube-api-access-tzz28\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.433193 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.433159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.433193 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.433187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.433303 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.433231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e37e222-5226-4118-ad77-048d2ca4db00-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.433645 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.433611    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e37e222-5226-4118-ad77-048d2ca4db00-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.433782 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.433761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.433852 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.433811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.435549 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.435529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e37e222-5226-4118-ad77-048d2ca4db00-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.442223 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.442204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzz28\" (UniqueName: \"kubernetes.io/projected/1e37e222-5226-4118-ad77-048d2ca4db00-kube-api-access-tzz28\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.530456 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.530424    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:47:57.800027 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:57.799987    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"]
Sep 18 20:47:57.801373 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:47:57.801344    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e37e222_5226_4118_ad77_048d2ca4db00.slice/crio-97feefcfbf476b5a1f35b9d9542bc4d3d789276af1f6779a94e43e7abd9d3fec WatchSource:0}: Error finding container 97feefcfbf476b5a1f35b9d9542bc4d3d789276af1f6779a94e43e7abd9d3fec: Status 404 returned error can't find the container with id 97feefcfbf476b5a1f35b9d9542bc4d3d789276af1f6779a94e43e7abd9d3fec
Sep 18 20:47:58.211066 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:58.210982    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" event={"ID":"1e37e222-5226-4118-ad77-048d2ca4db00","Type":"ContainerStarted","Data":"91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628"}
Sep 18 20:47:58.211201 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:47:58.211076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" event={"ID":"1e37e222-5226-4118-ad77-048d2ca4db00","Type":"ContainerStarted","Data":"97feefcfbf476b5a1f35b9d9542bc4d3d789276af1f6779a94e43e7abd9d3fec"}
Sep 18 20:48:03.232422 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:03.232390    2566 generic.go:356] "Generic (PLEG): container finished" podID="1e37e222-5226-4118-ad77-048d2ca4db00" containerID="91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628" exitCode=0
Sep 18 20:48:03.232803 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:03.232453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" event={"ID":"1e37e222-5226-4118-ad77-048d2ca4db00","Type":"ContainerDied","Data":"91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628"}
Sep 18 20:48:04.239110 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:04.239077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" event={"ID":"1e37e222-5226-4118-ad77-048d2ca4db00","Type":"ContainerStarted","Data":"209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8"}
Sep 18 20:48:04.239538 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:04.239137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" event={"ID":"1e37e222-5226-4118-ad77-048d2ca4db00","Type":"ContainerStarted","Data":"45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4"}
Sep 18 20:48:04.239538 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:04.239322    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:48:04.264227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:04.259983    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" podStartSLOduration=2.578646846 podStartE2EDuration="7.259964734s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.681317888s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:48:04.257623726 +0000 UTC m=+1653.288549867" watchObservedRunningTime="2026-09-18 20:48:04.259964734 +0000 UTC m=+1653.290890876"
Sep 18 20:48:05.243832 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:05.243796    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:48:05.245185 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:05.245163    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:48:06.247679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:06.247633    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:48:11.251898 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:11.251872    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:48:11.252369 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:11.252340    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 18 20:48:12.523502 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.523470    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"]
Sep 18 20:48:12.526944 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.526923    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.530039 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.529994    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 18 20:48:12.530156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.530001    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 18 20:48:12.537307 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.537282    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"]
Sep 18 20:48:12.555203 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.551679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.555203 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.551730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.555203 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.551850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.555203 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.552476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-proxy-tls\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.555203 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.552553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwxs\" (UniqueName: \"kubernetes.io/projected/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-api-access-gmwxs\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.653403 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.653363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-proxy-tls\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.653612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.653426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwxs\" (UniqueName: \"kubernetes.io/projected/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-api-access-gmwxs\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.653612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.653504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.653612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.653536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.653612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.653569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.654109 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.654085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.654367 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.654346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.654458 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.654370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.656176 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.656144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-proxy-tls\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.664147 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.664123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwxs\" (UniqueName: \"kubernetes.io/projected/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-api-access-gmwxs\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-r7f9c\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:12.838662 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:12.838570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:13.117350 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:13.117322    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"]
Sep 18 20:48:13.119822 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:48:13.119798    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65f15ca_f0c5_4d5c_9a67_c3e1112db877.slice/crio-bf58d76c63db6d9111a110d0b537808e2710ab3caef5dda7dc8b9a7bfaffd75a WatchSource:0}: Error finding container bf58d76c63db6d9111a110d0b537808e2710ab3caef5dda7dc8b9a7bfaffd75a: Status 404 returned error can't find the container with id bf58d76c63db6d9111a110d0b537808e2710ab3caef5dda7dc8b9a7bfaffd75a
Sep 18 20:48:13.277324 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:13.277289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" event={"ID":"a65f15ca-f0c5-4d5c-9a67-c3e1112db877","Type":"ContainerStarted","Data":"60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746"}
Sep 18 20:48:13.277496 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:13.277354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" event={"ID":"a65f15ca-f0c5-4d5c-9a67-c3e1112db877","Type":"ContainerStarted","Data":"bf58d76c63db6d9111a110d0b537808e2710ab3caef5dda7dc8b9a7bfaffd75a"}
Sep 18 20:48:17.296860 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:17.296830    2566 generic.go:356] "Generic (PLEG): container finished" podID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerID="60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746" exitCode=0
Sep 18 20:48:17.297246 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:17.296895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" event={"ID":"a65f15ca-f0c5-4d5c-9a67-c3e1112db877","Type":"ContainerDied","Data":"60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746"}
Sep 18 20:48:18.309395 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:18.309359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" event={"ID":"a65f15ca-f0c5-4d5c-9a67-c3e1112db877","Type":"ContainerStarted","Data":"dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb"}
Sep 18 20:48:18.309796 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:18.309442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" event={"ID":"a65f15ca-f0c5-4d5c-9a67-c3e1112db877","Type":"ContainerStarted","Data":"89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871"}
Sep 18 20:48:18.309796 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:18.309701    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:18.309928 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:18.309859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:18.311247 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:18.311221    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:48:18.329628 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:18.329587    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podStartSLOduration=2.439641047 podStartE2EDuration="6.329574228s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.889933181s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:48:18.326942954 +0000 UTC m=+1667.357869085" watchObservedRunningTime="2026-09-18 20:48:18.329574228 +0000 UTC m=+1667.360500368"
Sep 18 20:48:19.313679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:19.313644    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:48:20.317570 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:20.317523    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:48:21.253638 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:21.253605    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:48:25.323380 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:25.323350    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:48:25.323942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:25.323915    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:48:32.579634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:32.579594    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"]
Sep 18 20:48:32.580065 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:32.579897    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kserve-container" containerID="cri-o://45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4" gracePeriod=30
Sep 18 20:48:32.580065 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:32.579967    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kube-rbac-proxy" containerID="cri-o://209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8" gracePeriod=30
Sep 18 20:48:33.381031 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:33.380979    2566 generic.go:356] "Generic (PLEG): container finished" podID="1e37e222-5226-4118-ad77-048d2ca4db00" containerID="209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8" exitCode=2
Sep 18 20:48:33.381208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:33.381033    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" event={"ID":"1e37e222-5226-4118-ad77-048d2ca4db00","Type":"ContainerDied","Data":"209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8"}
Sep 18 20:48:33.918812 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:33.918790    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:48:34.029802 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.029774    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzz28\" (UniqueName: \"kubernetes.io/projected/1e37e222-5226-4118-ad77-048d2ca4db00-kube-api-access-tzz28\") pod \"1e37e222-5226-4118-ad77-048d2ca4db00\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") "
Sep 18 20:48:34.029972 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.029887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e37e222-5226-4118-ad77-048d2ca4db00-kserve-provision-location\") pod \"1e37e222-5226-4118-ad77-048d2ca4db00\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") "
Sep 18 20:48:34.029972 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.029927    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-cabundle-cert\") pod \"1e37e222-5226-4118-ad77-048d2ca4db00\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") "
Sep 18 20:48:34.029972 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.029951    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e37e222-5226-4118-ad77-048d2ca4db00-proxy-tls\") pod \"1e37e222-5226-4118-ad77-048d2ca4db00\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") "
Sep 18 20:48:34.030181 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.030032    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-kube-rbac-proxy-sar-config\") pod \"1e37e222-5226-4118-ad77-048d2ca4db00\" (UID: \"1e37e222-5226-4118-ad77-048d2ca4db00\") "
Sep 18 20:48:34.030328 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.030302    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e37e222-5226-4118-ad77-048d2ca4db00" (UID: "1e37e222-5226-4118-ad77-048d2ca4db00"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:48:34.030381 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.030309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e37e222-5226-4118-ad77-048d2ca4db00" (UID: "1e37e222-5226-4118-ad77-048d2ca4db00"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:48:34.030854 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.030835    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e37e222-5226-4118-ad77-048d2ca4db00-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e37e222-5226-4118-ad77-048d2ca4db00" (UID: "1e37e222-5226-4118-ad77-048d2ca4db00"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:48:34.031920 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.031904    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e37e222-5226-4118-ad77-048d2ca4db00-kube-api-access-tzz28" (OuterVolumeSpecName: "kube-api-access-tzz28") pod "1e37e222-5226-4118-ad77-048d2ca4db00" (UID: "1e37e222-5226-4118-ad77-048d2ca4db00"). InnerVolumeSpecName "kube-api-access-tzz28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:48:34.031976 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.031947    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e37e222-5226-4118-ad77-048d2ca4db00-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e37e222-5226-4118-ad77-048d2ca4db00" (UID: "1e37e222-5226-4118-ad77-048d2ca4db00"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:48:34.131213 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.131189    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e37e222-5226-4118-ad77-048d2ca4db00-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:48:34.131213 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.131209    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:48:34.131213 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.131219    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e37e222-5226-4118-ad77-048d2ca4db00-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:48:34.131415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.131227    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e37e222-5226-4118-ad77-048d2ca4db00-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:48:34.131415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.131235    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzz28\" (UniqueName: \"kubernetes.io/projected/1e37e222-5226-4118-ad77-048d2ca4db00-kube-api-access-tzz28\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:48:34.386779 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.386693    2566 generic.go:356] "Generic (PLEG): container finished" podID="1e37e222-5226-4118-ad77-048d2ca4db00" containerID="45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4" exitCode=0
Sep 18 20:48:34.386779 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.386745    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"
Sep 18 20:48:34.386953 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.386782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" event={"ID":"1e37e222-5226-4118-ad77-048d2ca4db00","Type":"ContainerDied","Data":"45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4"}
Sep 18 20:48:34.386953 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.386845    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8" event={"ID":"1e37e222-5226-4118-ad77-048d2ca4db00","Type":"ContainerDied","Data":"97feefcfbf476b5a1f35b9d9542bc4d3d789276af1f6779a94e43e7abd9d3fec"}
Sep 18 20:48:34.386953 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.386872    2566 scope.go:117] "RemoveContainer" containerID="209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8"
Sep 18 20:48:34.395814 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.395794    2566 scope.go:117] "RemoveContainer" containerID="45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4"
Sep 18 20:48:34.402969 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.402949    2566 scope.go:117] "RemoveContainer" containerID="91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628"
Sep 18 20:48:34.408634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.408609    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"]
Sep 18 20:48:34.410187 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.410171    2566 scope.go:117] "RemoveContainer" containerID="209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8"
Sep 18 20:48:34.410449 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:48:34.410416    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8\": container with ID starting with 209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8 not found: ID does not exist" containerID="209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8"
Sep 18 20:48:34.410651 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.410445    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8"} err="failed to get container status \"209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8\": rpc error: code = NotFound desc = could not find container \"209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8\": container with ID starting with 209f5fa49690adc928b752ad86a3ca9ca65b725c26bd1de783f9546fe864d0f8 not found: ID does not exist"
Sep 18 20:48:34.410651 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.410520    2566 scope.go:117] "RemoveContainer" containerID="45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4"
Sep 18 20:48:34.411267 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:48:34.411199    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4\": container with ID starting with 45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4 not found: ID does not exist" containerID="45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4"
Sep 18 20:48:34.411267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.411229    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4"} err="failed to get container status \"45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4\": rpc error: code = NotFound desc = could not find container \"45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4\": container with ID starting with 45c5d66d531556b887440e6ee285784f1509b5368dfe4dc90610701b2e592ae4 not found: ID does not exist"
Sep 18 20:48:34.411267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.411252    2566 scope.go:117] "RemoveContainer" containerID="91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628"
Sep 18 20:48:34.411625 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:48:34.411601    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628\": container with ID starting with 91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628 not found: ID does not exist" containerID="91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628"
Sep 18 20:48:34.411674 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.411635    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628"} err="failed to get container status \"91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628\": rpc error: code = NotFound desc = could not find container \"91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628\": container with ID starting with 91e5258aa24679b36aff182c26203c4cce25b0f51d17a21113ae5c75bcb62628 not found: ID does not exist"
Sep 18 20:48:34.412447 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:34.412430    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-c72x8"]
Sep 18 20:48:35.324160 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:35.324114    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:48:35.465195 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:35.465138    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" path="/var/lib/kubelet/pods/1e37e222-5226-4118-ad77-048d2ca4db00/volumes"
Sep 18 20:48:37.792973 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.792939    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"]
Sep 18 20:48:37.793383 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.793368    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kserve-container"
Sep 18 20:48:37.793426 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.793386    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kserve-container"
Sep 18 20:48:37.793426 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.793411    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="storage-initializer"
Sep 18 20:48:37.793426 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.793419    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="storage-initializer"
Sep 18 20:48:37.793521 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.793441    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kube-rbac-proxy"
Sep 18 20:48:37.793521 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.793450    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kube-rbac-proxy"
Sep 18 20:48:37.793621 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.793523    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kube-rbac-proxy"
Sep 18 20:48:37.793621 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.793535    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e37e222-5226-4118-ad77-048d2ca4db00" containerName="kserve-container"
Sep 18 20:48:37.798608 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.798586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:37.800975 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.800945    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 20:48:37.801117 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.801034    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 18 20:48:37.807608 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.807584    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"]
Sep 18 20:48:37.828939 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.828915    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"]
Sep 18 20:48:37.832074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.832058    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:37.834390 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.834372    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 20:48:37.842889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.842869    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"]
Sep 18 20:48:37.963161 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:37.963161 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:37.963340 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls6gh\" (UniqueName: \"kubernetes.io/projected/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-api-access-ls6gh\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:37.963340 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:37.963340 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hpq\" (UniqueName: \"kubernetes.io/projected/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-api-access-n8hpq\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:37.963340 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963324    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:37.963506 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:37.963506 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:37.963506 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3f3488-c5b2-4290-8724-88fdf6a2f472-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:37.963506 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:37.963421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.064575 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.064498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.064575 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.064544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.064753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.064592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls6gh\" (UniqueName: \"kubernetes.io/projected/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-api-access-ls6gh\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.064753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.064643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.064753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.064682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hpq\" (UniqueName: \"kubernetes.io/projected/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-api-access-n8hpq\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.064753 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.064705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.064993 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.064961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.065202 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.065031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.065202 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.065060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3f3488-c5b2-4290-8724-88fdf6a2f472-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.065202 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.065089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.065429 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:48:38.065299    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:48:38.065429 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:48:38.065377    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3f3488-c5b2-4290-8724-88fdf6a2f472-proxy-tls podName:0a3f3488-c5b2-4290-8724-88fdf6a2f472 nodeName:}" failed. No retries permitted until 2026-09-18 20:48:38.565357085 +0000 UTC m=+1687.596283203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0a3f3488-c5b2-4290-8724-88fdf6a2f472-proxy-tls") pod "isvc-canary-create-predictor-76b74f4d6c-mspk8" (UID: "0a3f3488-c5b2-4290-8724-88fdf6a2f472") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:48:38.065429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.065299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.065610 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.065478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.065610 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.065499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.065610 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.065569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.065765 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.065750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.065896 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.065875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.067946 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.067925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.075819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.075793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hpq\" (UniqueName: \"kubernetes.io/projected/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-api-access-n8hpq\") pod \"isvc-canary-create-v2-predictor-667f95d746-dg4nq\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.075819 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.075808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls6gh\" (UniqueName: \"kubernetes.io/projected/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-api-access-ls6gh\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.110545 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.110526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:38.384431 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.384353    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"]
Sep 18 20:48:38.387005 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:48:38.386976    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90f98ca_08d7_4caa_ac90_2f4f2c8415c9.slice/crio-b2de69f01559c09ce3438011e12d9929cb4a91a770a0b686797fd0c48cd50bb7 WatchSource:0}: Error finding container b2de69f01559c09ce3438011e12d9929cb4a91a770a0b686797fd0c48cd50bb7: Status 404 returned error can't find the container with id b2de69f01559c09ce3438011e12d9929cb4a91a770a0b686797fd0c48cd50bb7
Sep 18 20:48:38.405080 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.405049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" event={"ID":"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9","Type":"ContainerStarted","Data":"b2de69f01559c09ce3438011e12d9929cb4a91a770a0b686797fd0c48cd50bb7"}
Sep 18 20:48:38.569949 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.569909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3f3488-c5b2-4290-8724-88fdf6a2f472-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.572208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.572186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3f3488-c5b2-4290-8724-88fdf6a2f472-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-mspk8\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:38.742313 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:38.742226    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:39.013511 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:39.013427    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"]
Sep 18 20:48:39.016160 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:48:39.016128    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3f3488_c5b2_4290_8724_88fdf6a2f472.slice/crio-cc2ce3623fa35a86a55963e358c87e4b5bafc9067d25d38e723d71822965a0c1 WatchSource:0}: Error finding container cc2ce3623fa35a86a55963e358c87e4b5bafc9067d25d38e723d71822965a0c1: Status 404 returned error can't find the container with id cc2ce3623fa35a86a55963e358c87e4b5bafc9067d25d38e723d71822965a0c1
Sep 18 20:48:39.412283 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:39.412239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" event={"ID":"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9","Type":"ContainerStarted","Data":"e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664"}
Sep 18 20:48:39.413887 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:39.413861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" event={"ID":"0a3f3488-c5b2-4290-8724-88fdf6a2f472","Type":"ContainerStarted","Data":"28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d"}
Sep 18 20:48:39.414054 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:39.413909    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" event={"ID":"0a3f3488-c5b2-4290-8724-88fdf6a2f472","Type":"ContainerStarted","Data":"cc2ce3623fa35a86a55963e358c87e4b5bafc9067d25d38e723d71822965a0c1"}
Sep 18 20:48:42.427491 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:42.427449    2566 generic.go:356] "Generic (PLEG): container finished" podID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerID="e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664" exitCode=0
Sep 18 20:48:42.427879 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:42.427519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" event={"ID":"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9","Type":"ContainerDied","Data":"e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664"}
Sep 18 20:48:43.433158 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:43.433122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" event={"ID":"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9","Type":"ContainerStarted","Data":"2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4"}
Sep 18 20:48:43.433574 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:43.433189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" event={"ID":"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9","Type":"ContainerStarted","Data":"d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe"}
Sep 18 20:48:43.433574 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:43.433482    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:43.433574 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:43.433547    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:43.434671 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:43.434642    2566 generic.go:356] "Generic (PLEG): container finished" podID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerID="28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d" exitCode=0
Sep 18 20:48:43.434755 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:43.434705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" event={"ID":"0a3f3488-c5b2-4290-8724-88fdf6a2f472","Type":"ContainerDied","Data":"28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d"}
Sep 18 20:48:43.434910 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:43.434865    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:48:43.452357 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:43.452258    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podStartSLOduration=2.623567226 podStartE2EDuration="6.452241707s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.828674481s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:48:43.45126648 +0000 UTC m=+1692.482192621" watchObservedRunningTime="2026-09-18 20:48:43.452241707 +0000 UTC m=+1692.483167850"
Sep 18 20:48:44.441985 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:44.441947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" event={"ID":"0a3f3488-c5b2-4290-8724-88fdf6a2f472","Type":"ContainerStarted","Data":"ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7"}
Sep 18 20:48:44.442389 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:44.442036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" event={"ID":"0a3f3488-c5b2-4290-8724-88fdf6a2f472","Type":"ContainerStarted","Data":"ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593"}
Sep 18 20:48:44.442389 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:44.442266    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:48:44.442516 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:44.442496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:44.442579 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:44.442556    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:44.443758 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:44.443736    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:48:44.461910 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:44.461857    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podStartSLOduration=3.7736079240000002 podStartE2EDuration="7.461843724s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.6882358s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:48:44.460047704 +0000 UTC m=+1693.490973844" watchObservedRunningTime="2026-09-18 20:48:44.461843724 +0000 UTC m=+1693.492769865"
Sep 18 20:48:45.324425 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:45.324389    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:48:45.446605 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:45.446563    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:48:49.446548 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:49.446514    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:48:49.447093 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:49.447064    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:48:50.451624 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:50.451591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:48:50.452196 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:50.452169    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:48:55.324139 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:55.324053    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:48:59.447904 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:48:59.447866    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:49:00.452352 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:00.452307    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:49:05.324340 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:05.324300    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:49:09.447381 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:09.447341    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:49:10.453031 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:10.452986    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:49:15.324793 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:15.324749    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:49:19.447341 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:19.447304    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:49:20.452301 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:20.452264    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:49:25.324817 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:25.324774    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:49:29.447707 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:29.447668    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:49:30.452098 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:30.452058    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:49:35.325079 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:35.325041    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:49:39.447442 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:39.447396    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:49:40.452212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:40.452177    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:49:43.329820 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:43.329790    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"]
Sep 18 20:49:43.330253 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:43.330030    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" containerID="cri-o://89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871" gracePeriod=30
Sep 18 20:49:43.330253 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:43.330078    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kube-rbac-proxy" containerID="cri-o://dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb" gracePeriod=30
Sep 18 20:49:43.680927 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:43.680831    2566 generic.go:356] "Generic (PLEG): container finished" podID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerID="dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb" exitCode=2
Sep 18 20:49:43.680927 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:43.680900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" event={"ID":"a65f15ca-f0c5-4d5c-9a67-c3e1112db877","Type":"ContainerDied","Data":"dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb"}
Sep 18 20:49:45.317919 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:45.317883    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 18 20:49:45.323843 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:45.323820    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 18 20:49:48.163086 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.163058    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:49:48.245482 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.245408    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-cabundle-cert\") pod \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") "
Sep 18 20:49:48.245482 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.245438    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmwxs\" (UniqueName: \"kubernetes.io/projected/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-api-access-gmwxs\") pod \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") "
Sep 18 20:49:48.245482 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.245482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kserve-provision-location\") pod \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") "
Sep 18 20:49:48.245745 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.245506    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-rbac-proxy-sar-config\") pod \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") "
Sep 18 20:49:48.245745 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.245528    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-proxy-tls\") pod \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\" (UID: \"a65f15ca-f0c5-4d5c-9a67-c3e1112db877\") "
Sep 18 20:49:48.245863 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.245820    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a65f15ca-f0c5-4d5c-9a67-c3e1112db877" (UID: "a65f15ca-f0c5-4d5c-9a67-c3e1112db877"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:49:48.245910 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.245851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a65f15ca-f0c5-4d5c-9a67-c3e1112db877" (UID: "a65f15ca-f0c5-4d5c-9a67-c3e1112db877"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:49:48.245910 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.245864    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a65f15ca-f0c5-4d5c-9a67-c3e1112db877" (UID: "a65f15ca-f0c5-4d5c-9a67-c3e1112db877"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:49:48.247483 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.247465    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a65f15ca-f0c5-4d5c-9a67-c3e1112db877" (UID: "a65f15ca-f0c5-4d5c-9a67-c3e1112db877"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:49:48.247550 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.247533    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-api-access-gmwxs" (OuterVolumeSpecName: "kube-api-access-gmwxs") pod "a65f15ca-f0c5-4d5c-9a67-c3e1112db877" (UID: "a65f15ca-f0c5-4d5c-9a67-c3e1112db877"). InnerVolumeSpecName "kube-api-access-gmwxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:49:48.346272 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.346247    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:49:48.346272 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.346270    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:49:48.346428 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.346281    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:49:48.346428 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.346290    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmwxs\" (UniqueName: \"kubernetes.io/projected/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kube-api-access-gmwxs\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:49:48.346428 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.346298    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65f15ca-f0c5-4d5c-9a67-c3e1112db877-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:49:48.589559 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.589526    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"]
Sep 18 20:49:48.589856 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.589844    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container"
Sep 18 20:49:48.589901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.589858    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container"
Sep 18 20:49:48.589901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.589869    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="storage-initializer"
Sep 18 20:49:48.589901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.589875    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="storage-initializer"
Sep 18 20:49:48.589901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.589882    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kube-rbac-proxy"
Sep 18 20:49:48.589901 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.589891    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kube-rbac-proxy"
Sep 18 20:49:48.590072 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.589943    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kserve-container"
Sep 18 20:49:48.590072 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.589950    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerName="kube-rbac-proxy"
Sep 18 20:49:48.593137 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.593118    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.595907 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.595860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 18 20:49:48.596242 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.596223    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 18 20:49:48.604429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.604409    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"]
Sep 18 20:49:48.648765 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.648738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.648892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.648778    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.648892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.648814    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5558883-e8ba-4c7f-ad25-d38256e2e31a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.648892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.648839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6j9r\" (UniqueName: \"kubernetes.io/projected/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-api-access-s6j9r\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.648892 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.648865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.702470 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.702444    2566 generic.go:356] "Generic (PLEG): container finished" podID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" containerID="89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871" exitCode=0
Sep 18 20:49:48.702612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.702492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" event={"ID":"a65f15ca-f0c5-4d5c-9a67-c3e1112db877","Type":"ContainerDied","Data":"89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871"}
Sep 18 20:49:48.702612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.702539    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"
Sep 18 20:49:48.702612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.702545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c" event={"ID":"a65f15ca-f0c5-4d5c-9a67-c3e1112db877","Type":"ContainerDied","Data":"bf58d76c63db6d9111a110d0b537808e2710ab3caef5dda7dc8b9a7bfaffd75a"}
Sep 18 20:49:48.702612 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.702572    2566 scope.go:117] "RemoveContainer" containerID="dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb"
Sep 18 20:49:48.715540 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.715514    2566 scope.go:117] "RemoveContainer" containerID="89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871"
Sep 18 20:49:48.724400 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.724384    2566 scope.go:117] "RemoveContainer" containerID="60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746"
Sep 18 20:49:48.731571 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.731550    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"]
Sep 18 20:49:48.731877 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.731862    2566 scope.go:117] "RemoveContainer" containerID="dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb"
Sep 18 20:49:48.732149 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:49:48.732132    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb\": container with ID starting with dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb not found: ID does not exist" containerID="dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb"
Sep 18 20:49:48.732211 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.732156    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb"} err="failed to get container status \"dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb\": rpc error: code = NotFound desc = could not find container \"dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb\": container with ID starting with dfab9103696c4e53d2337c8b0df2c75ab1da4861625cc8610d43279de3cd72cb not found: ID does not exist"
Sep 18 20:49:48.732211 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.732173    2566 scope.go:117] "RemoveContainer" containerID="89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871"
Sep 18 20:49:48.732374 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:49:48.732359    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871\": container with ID starting with 89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871 not found: ID does not exist" containerID="89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871"
Sep 18 20:49:48.732418 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.732378    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871"} err="failed to get container status \"89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871\": rpc error: code = NotFound desc = could not find container \"89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871\": container with ID starting with 89436618b79e795d92f25795dbd5557055109dfae5c030c9cfe307adb1ad5871 not found: ID does not exist"
Sep 18 20:49:48.732418 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.732392    2566 scope.go:117] "RemoveContainer" containerID="60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746"
Sep 18 20:49:48.732603 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:49:48.732589    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746\": container with ID starting with 60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746 not found: ID does not exist" containerID="60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746"
Sep 18 20:49:48.732640 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.732605    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746"} err="failed to get container status \"60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746\": rpc error: code = NotFound desc = could not find container \"60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746\": container with ID starting with 60c119b51a03ebf5904c6b3c19480a3d19d6f5286abf72e2e5c1337aab938746 not found: ID does not exist"
Sep 18 20:49:48.734929 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.734908    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-r7f9c"]
Sep 18 20:49:48.749341 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.749322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5558883-e8ba-4c7f-ad25-d38256e2e31a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.749451 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.749348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6j9r\" (UniqueName: \"kubernetes.io/projected/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-api-access-s6j9r\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.749451 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.749374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.749451 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.749403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.749451 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.749438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.749663 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:49:48.749474    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 18 20:49:48.749663 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:49:48.749546    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5558883-e8ba-4c7f-ad25-d38256e2e31a-proxy-tls podName:a5558883-e8ba-4c7f-ad25-d38256e2e31a nodeName:}" failed. No retries permitted until 2026-09-18 20:49:49.249522965 +0000 UTC m=+1758.280449100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a5558883-e8ba-4c7f-ad25-d38256e2e31a-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" (UID: "a5558883-e8ba-4c7f-ad25-d38256e2e31a") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 18 20:49:48.749812 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.749792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.750097 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.750078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.750140 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.750087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:48.758549 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:48.758529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6j9r\" (UniqueName: \"kubernetes.io/projected/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-api-access-s6j9r\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:49.254196 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:49.254163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5558883-e8ba-4c7f-ad25-d38256e2e31a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:49.256521 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:49.256501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5558883-e8ba-4c7f-ad25-d38256e2e31a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:49.446948 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:49.446904    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:49:49.459716 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:49.459685    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a65f15ca-f0c5-4d5c-9a67-c3e1112db877" path="/var/lib/kubelet/pods/a65f15ca-f0c5-4d5c-9a67-c3e1112db877/volumes"
Sep 18 20:49:49.504762 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:49.504698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:49.778089 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:49.778064    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"]
Sep 18 20:49:49.780342 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:49:49.780319    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5558883_e8ba_4c7f_ad25_d38256e2e31a.slice/crio-8f6a1da7603471879082600c446b2d62301369c6f22daedfdf2b6e08cbe1dda4 WatchSource:0}: Error finding container 8f6a1da7603471879082600c446b2d62301369c6f22daedfdf2b6e08cbe1dda4: Status 404 returned error can't find the container with id 8f6a1da7603471879082600c446b2d62301369c6f22daedfdf2b6e08cbe1dda4
Sep 18 20:49:50.452977 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:50.452938    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 18 20:49:50.712613 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:50.712530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" event={"ID":"a5558883-e8ba-4c7f-ad25-d38256e2e31a","Type":"ContainerStarted","Data":"a18da3a0c5dcfcd72022101af16a3e7904939ae4b054e92aba319f748f6a3c62"}
Sep 18 20:49:50.712613 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:50.712595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" event={"ID":"a5558883-e8ba-4c7f-ad25-d38256e2e31a","Type":"ContainerStarted","Data":"8f6a1da7603471879082600c446b2d62301369c6f22daedfdf2b6e08cbe1dda4"}
Sep 18 20:49:53.726995 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:53.726906    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerID="a18da3a0c5dcfcd72022101af16a3e7904939ae4b054e92aba319f748f6a3c62" exitCode=0
Sep 18 20:49:53.726995 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:53.726980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" event={"ID":"a5558883-e8ba-4c7f-ad25-d38256e2e31a","Type":"ContainerDied","Data":"a18da3a0c5dcfcd72022101af16a3e7904939ae4b054e92aba319f748f6a3c62"}
Sep 18 20:49:54.740795 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:54.740761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" event={"ID":"a5558883-e8ba-4c7f-ad25-d38256e2e31a","Type":"ContainerStarted","Data":"5502264904c27b1357d30dc9cf742027c6664bf12677639bf937de9ed5aa4434"}
Sep 18 20:49:54.741192 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:54.740844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" event={"ID":"a5558883-e8ba-4c7f-ad25-d38256e2e31a","Type":"ContainerStarted","Data":"c9bdd8eb18e6ddfefbb724db58e17792d60edaec03308711e98001a12cea7e28"}
Sep 18 20:49:54.741192 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:54.741050    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:54.760368 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:54.760325    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podStartSLOduration=3.262777656 podStartE2EDuration="6.760309466s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.49753181s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:49:54.758987854 +0000 UTC m=+1763.789913994" watchObservedRunningTime="2026-09-18 20:49:54.760309466 +0000 UTC m=+1763.791235669"
Sep 18 20:49:55.744993 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:55.744966    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:49:55.746256 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:55.746224    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:49:56.749826 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:56.749787    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:49:59.448166 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:59.448136    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:49:59.574460 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:59.574427    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"]
Sep 18 20:49:59.574667 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:59.574644    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container" containerID="cri-o://ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593" gracePeriod=30
Sep 18 20:49:59.574744 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:59.574716    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kube-rbac-proxy" containerID="cri-o://ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7" gracePeriod=30
Sep 18 20:49:59.575641 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:59.575623    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:49:59.763455 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:59.763378    2566 generic.go:356] "Generic (PLEG): container finished" podID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerID="ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7" exitCode=2
Sep 18 20:49:59.763455 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:49:59.763416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" event={"ID":"0a3f3488-c5b2-4290-8724-88fdf6a2f472","Type":"ContainerDied","Data":"ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7"}
Sep 18 20:50:00.447636 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:00.447595    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 18 20:50:01.753993 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:01.753964    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:50:01.754582 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:01.754560    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:50:04.421104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.421083    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:50:04.483642 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.483614    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kserve-provision-location\") pod \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") "
Sep 18 20:50:04.483804 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.483671    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-rbac-proxy-sar-config\") pod \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") "
Sep 18 20:50:04.483804 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.483696    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-cabundle-cert\") pod \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") "
Sep 18 20:50:04.483804 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.483730    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls6gh\" (UniqueName: \"kubernetes.io/projected/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-api-access-ls6gh\") pod \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") "
Sep 18 20:50:04.483804 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.483761    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3f3488-c5b2-4290-8724-88fdf6a2f472-proxy-tls\") pod \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\" (UID: \"0a3f3488-c5b2-4290-8724-88fdf6a2f472\") "
Sep 18 20:50:04.484033 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.483927    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a3f3488-c5b2-4290-8724-88fdf6a2f472" (UID: "0a3f3488-c5b2-4290-8724-88fdf6a2f472"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:50:04.484103 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.484055    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a3f3488-c5b2-4290-8724-88fdf6a2f472" (UID: "0a3f3488-c5b2-4290-8724-88fdf6a2f472"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:50:04.484161 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.484107    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a3f3488-c5b2-4290-8724-88fdf6a2f472" (UID: "0a3f3488-c5b2-4290-8724-88fdf6a2f472"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:50:04.485743 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.485717    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3f3488-c5b2-4290-8724-88fdf6a2f472-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a3f3488-c5b2-4290-8724-88fdf6a2f472" (UID: "0a3f3488-c5b2-4290-8724-88fdf6a2f472"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:50:04.485874 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.485857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-api-access-ls6gh" (OuterVolumeSpecName: "kube-api-access-ls6gh") pod "0a3f3488-c5b2-4290-8724-88fdf6a2f472" (UID: "0a3f3488-c5b2-4290-8724-88fdf6a2f472"). InnerVolumeSpecName "kube-api-access-ls6gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:50:04.584700 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.584618    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:04.584700 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.584648    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:04.584700 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.584663    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a3f3488-c5b2-4290-8724-88fdf6a2f472-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:04.584700 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.584675    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls6gh\" (UniqueName: \"kubernetes.io/projected/0a3f3488-c5b2-4290-8724-88fdf6a2f472-kube-api-access-ls6gh\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:04.584700 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.584687    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a3f3488-c5b2-4290-8724-88fdf6a2f472-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:04.787866 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.787830    2566 generic.go:356] "Generic (PLEG): container finished" podID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerID="ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593" exitCode=0
Sep 18 20:50:04.788060 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.787895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" event={"ID":"0a3f3488-c5b2-4290-8724-88fdf6a2f472","Type":"ContainerDied","Data":"ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593"}
Sep 18 20:50:04.788060 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.787938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8" event={"ID":"0a3f3488-c5b2-4290-8724-88fdf6a2f472","Type":"ContainerDied","Data":"cc2ce3623fa35a86a55963e358c87e4b5bafc9067d25d38e723d71822965a0c1"}
Sep 18 20:50:04.788060 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.787965    2566 scope.go:117] "RemoveContainer" containerID="ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7"
Sep 18 20:50:04.788060 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.787897    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"
Sep 18 20:50:04.796212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.796194    2566 scope.go:117] "RemoveContainer" containerID="ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593"
Sep 18 20:50:04.808872 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.808846    2566 scope.go:117] "RemoveContainer" containerID="28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d"
Sep 18 20:50:04.811792 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.811769    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"]
Sep 18 20:50:04.816163 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.816144    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-mspk8"]
Sep 18 20:50:04.816652 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.816632    2566 scope.go:117] "RemoveContainer" containerID="ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7"
Sep 18 20:50:04.816906 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:50:04.816889    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7\": container with ID starting with ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7 not found: ID does not exist" containerID="ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7"
Sep 18 20:50:04.816957 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.816913    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7"} err="failed to get container status \"ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7\": rpc error: code = NotFound desc = could not find container \"ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7\": container with ID starting with ddf7784f7af43ef1f7ca8fb70e0b448ff27fcb795d5b7408b49c1ed45f6d36d7 not found: ID does not exist"
Sep 18 20:50:04.816957 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.816930    2566 scope.go:117] "RemoveContainer" containerID="ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593"
Sep 18 20:50:04.817278 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:50:04.817260    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593\": container with ID starting with ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593 not found: ID does not exist" containerID="ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593"
Sep 18 20:50:04.817344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.817281    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593"} err="failed to get container status \"ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593\": rpc error: code = NotFound desc = could not find container \"ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593\": container with ID starting with ec231b6db573ea83a74b4a22af992d0b6d32fed489e3ba7f6cb8159cb93b3593 not found: ID does not exist"
Sep 18 20:50:04.817344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.817296    2566 scope.go:117] "RemoveContainer" containerID="28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d"
Sep 18 20:50:04.817532 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:50:04.817513    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d\": container with ID starting with 28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d not found: ID does not exist" containerID="28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d"
Sep 18 20:50:04.817573 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:04.817539    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d"} err="failed to get container status \"28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d\": rpc error: code = NotFound desc = could not find container \"28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d\": container with ID starting with 28bd0d6e2820029957aaa47ed8c82bf715afeca13a2eca4c07fe69506306bf7d not found: ID does not exist"
Sep 18 20:50:05.460387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:05.460352    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" path="/var/lib/kubelet/pods/0a3f3488-c5b2-4290-8724-88fdf6a2f472/volumes"
Sep 18 20:50:11.754564 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:11.754527    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:50:17.854720 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:17.854692    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"]
Sep 18 20:50:17.855203 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:17.854930    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" containerID="cri-o://d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe" gracePeriod=30
Sep 18 20:50:17.855203 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:17.854954    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kube-rbac-proxy" containerID="cri-o://2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4" gracePeriod=30
Sep 18 20:50:18.852536 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:18.852501    2566 generic.go:356] "Generic (PLEG): container finished" podID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerID="2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4" exitCode=2
Sep 18 20:50:18.852707 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:18.852570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" event={"ID":"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9","Type":"ContainerDied","Data":"2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4"}
Sep 18 20:50:19.442759 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:19.442721    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 18 20:50:19.447038 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:19.446995    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 18 20:50:21.755042 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:21.754993    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:50:22.601260 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.601235    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:50:22.726155 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.726073    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kserve-provision-location\") pod \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") "
Sep 18 20:50:22.726155 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.726120    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-cabundle-cert\") pod \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") "
Sep 18 20:50:22.726155 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.726158    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8hpq\" (UniqueName: \"kubernetes.io/projected/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-api-access-n8hpq\") pod \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") "
Sep 18 20:50:22.726425 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.726188    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-rbac-proxy-sar-config\") pod \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") "
Sep 18 20:50:22.726425 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.726255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-proxy-tls\") pod \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\" (UID: \"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9\") "
Sep 18 20:50:22.726536 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.726507    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" (UID: "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:50:22.726536 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.726512    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" (UID: "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:50:22.726650 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.726615    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" (UID: "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:50:22.728486 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.728456    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" (UID: "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:50:22.728594 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.728547    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-api-access-n8hpq" (OuterVolumeSpecName: "kube-api-access-n8hpq") pod "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" (UID: "c90f98ca-08d7-4caa-ac90-2f4f2c8415c9"). InnerVolumeSpecName "kube-api-access-n8hpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:50:22.827524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.827485    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.827524 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.827523    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.827932 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.827538    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.827932 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.827553    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8hpq\" (UniqueName: \"kubernetes.io/projected/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-api-access-n8hpq\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.827932 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.827566    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:50:22.872553 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.872512    2566 generic.go:356] "Generic (PLEG): container finished" podID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerID="d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe" exitCode=0
Sep 18 20:50:22.872729 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.872600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" event={"ID":"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9","Type":"ContainerDied","Data":"d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe"}
Sep 18 20:50:22.872729 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.872617    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"
Sep 18 20:50:22.872729 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.872720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq" event={"ID":"c90f98ca-08d7-4caa-ac90-2f4f2c8415c9","Type":"ContainerDied","Data":"b2de69f01559c09ce3438011e12d9929cb4a91a770a0b686797fd0c48cd50bb7"}
Sep 18 20:50:22.872854 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.872767    2566 scope.go:117] "RemoveContainer" containerID="2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4"
Sep 18 20:50:22.881564 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.881544    2566 scope.go:117] "RemoveContainer" containerID="d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe"
Sep 18 20:50:22.890774 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.890760    2566 scope.go:117] "RemoveContainer" containerID="e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664"
Sep 18 20:50:22.895466 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.895442    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"]
Sep 18 20:50:22.899755 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.899651    2566 scope.go:117] "RemoveContainer" containerID="2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4"
Sep 18 20:50:22.899998 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:50:22.899975    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4\": container with ID starting with 2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4 not found: ID does not exist" containerID="2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4"
Sep 18 20:50:22.900156 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.900130    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4"} err="failed to get container status \"2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4\": rpc error: code = NotFound desc = could not find container \"2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4\": container with ID starting with 2867a01232d9692c9523897eed5d441fec50d55104086435b30d406038e7adc4 not found: ID does not exist"
Sep 18 20:50:22.900248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.900159    2566 scope.go:117] "RemoveContainer" containerID="d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe"
Sep 18 20:50:22.900560 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:50:22.900533    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe\": container with ID starting with d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe not found: ID does not exist" containerID="d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe"
Sep 18 20:50:22.900679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.900568    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe"} err="failed to get container status \"d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe\": rpc error: code = NotFound desc = could not find container \"d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe\": container with ID starting with d3d69b618e9afdd818ff26256de800ad30524cf065d5689f21ce963dd1164afe not found: ID does not exist"
Sep 18 20:50:22.900679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.900592    2566 scope.go:117] "RemoveContainer" containerID="e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664"
Sep 18 20:50:22.900679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.900601    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-dg4nq"]
Sep 18 20:50:22.900895 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:50:22.900877    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664\": container with ID starting with e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664 not found: ID does not exist" containerID="e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664"
Sep 18 20:50:22.900974 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:22.900901    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664"} err="failed to get container status \"e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664\": rpc error: code = NotFound desc = could not find container \"e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664\": container with ID starting with e54d5bd29948647c1221aabaab515dbdf7984cfb20f51fd5a55f1fd1b39e0664 not found: ID does not exist"
Sep 18 20:50:23.083800 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.083768    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"]
Sep 18 20:50:23.084209 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084195    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kube-rbac-proxy"
Sep 18 20:50:23.084262 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084213    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kube-rbac-proxy"
Sep 18 20:50:23.084262 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084226    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container"
Sep 18 20:50:23.084262 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084234    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container"
Sep 18 20:50:23.084262 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084249    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container"
Sep 18 20:50:23.084262 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084259    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container"
Sep 18 20:50:23.084415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084280    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kube-rbac-proxy"
Sep 18 20:50:23.084415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084288    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kube-rbac-proxy"
Sep 18 20:50:23.084415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084300    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="storage-initializer"
Sep 18 20:50:23.084415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084309    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="storage-initializer"
Sep 18 20:50:23.084415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084321    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="storage-initializer"
Sep 18 20:50:23.084415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084329    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="storage-initializer"
Sep 18 20:50:23.084415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084399    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kube-rbac-proxy"
Sep 18 20:50:23.084415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084410    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kserve-container"
Sep 18 20:50:23.084687 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084421    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" containerName="kserve-container"
Sep 18 20:50:23.084687 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.084432    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3f3488-c5b2-4290-8724-88fdf6a2f472" containerName="kube-rbac-proxy"
Sep 18 20:50:23.089283 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.089261    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.091811 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.091792    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 20:50:23.091940 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.091823    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:50:23.099224 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.099200    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"]
Sep 18 20:50:23.230371 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.230336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3b5146-78b4-45e7-a113-d5ef12558d43-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.230537 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.230377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjj8m\" (UniqueName: \"kubernetes.io/projected/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-api-access-bjj8m\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.230537 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.230409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3b5146-78b4-45e7-a113-d5ef12558d43-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.230537 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.230452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.230537 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.230474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.331452 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.331414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3b5146-78b4-45e7-a113-d5ef12558d43-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.331610 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.331472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.331610 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.331512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.331610 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.331582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3b5146-78b4-45e7-a113-d5ef12558d43-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.331757 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.331617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjj8m\" (UniqueName: \"kubernetes.io/projected/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-api-access-bjj8m\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.331869 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.331849    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3b5146-78b4-45e7-a113-d5ef12558d43-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.332176 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.332152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.332260 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.332246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.334053 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.334001    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3b5146-78b4-45e7-a113-d5ef12558d43-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.339052 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.339030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjj8m\" (UniqueName: \"kubernetes.io/projected/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-api-access-bjj8m\") pod \"isvc-canary-promote-predictor-6546f8697f-m5krp\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.400696 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.400669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:23.462043 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.462000    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c90f98ca-08d7-4caa-ac90-2f4f2c8415c9" path="/var/lib/kubelet/pods/c90f98ca-08d7-4caa-ac90-2f4f2c8415c9/volumes"
Sep 18 20:50:23.673463 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.673429    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"]
Sep 18 20:50:23.674189 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:50:23.674151    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3b5146_78b4_45e7_a113_d5ef12558d43.slice/crio-5325142d49f8e483b5854c23cfe88dd06e5c32a24b3fcff9abaff1635938889a WatchSource:0}: Error finding container 5325142d49f8e483b5854c23cfe88dd06e5c32a24b3fcff9abaff1635938889a: Status 404 returned error can't find the container with id 5325142d49f8e483b5854c23cfe88dd06e5c32a24b3fcff9abaff1635938889a
Sep 18 20:50:23.880440 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.880404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" event={"ID":"2a3b5146-78b4-45e7-a113-d5ef12558d43","Type":"ContainerStarted","Data":"49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4"}
Sep 18 20:50:23.880792 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:23.880461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" event={"ID":"2a3b5146-78b4-45e7-a113-d5ef12558d43","Type":"ContainerStarted","Data":"5325142d49f8e483b5854c23cfe88dd06e5c32a24b3fcff9abaff1635938889a"}
Sep 18 20:50:27.898054 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:27.897993    2566 generic.go:356] "Generic (PLEG): container finished" podID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerID="49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4" exitCode=0
Sep 18 20:50:27.898417 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:27.898070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" event={"ID":"2a3b5146-78b4-45e7-a113-d5ef12558d43","Type":"ContainerDied","Data":"49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4"}
Sep 18 20:50:28.904056 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:28.903997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" event={"ID":"2a3b5146-78b4-45e7-a113-d5ef12558d43","Type":"ContainerStarted","Data":"4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430"}
Sep 18 20:50:28.904424 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:28.904091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" event={"ID":"2a3b5146-78b4-45e7-a113-d5ef12558d43","Type":"ContainerStarted","Data":"4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29"}
Sep 18 20:50:28.904424 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:28.904361    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:28.904528 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:28.904511    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:28.905868 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:28.905841    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:50:28.923843 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:28.923798    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podStartSLOduration=2.341602895 podStartE2EDuration="5.923787196s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.582184301s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:50:28.922664613 +0000 UTC m=+1797.953590753" watchObservedRunningTime="2026-09-18 20:50:28.923787196 +0000 UTC m=+1797.954713335"
Sep 18 20:50:29.907810 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:29.907771    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:50:31.473509 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:31.473480    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:50:31.482809 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:31.482782    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:50:31.755291 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:31.755206    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:50:34.912549 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:34.912520    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:50:34.913081 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:34.913032    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:50:41.755286 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:41.755244    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:50:44.912964 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:44.912927    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:50:51.754699 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:51.754656    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:50:54.913251 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:50:54.913211    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:51:01.754804 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:01.754763    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:51:04.913123 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:04.913088    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:51:11.755034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:11.754986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:51:14.913481 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:14.913442    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:51:19.279312 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:19.279278    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"]
Sep 18 20:51:19.279754 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:19.279537    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" containerID="cri-o://c9bdd8eb18e6ddfefbb724db58e17792d60edaec03308711e98001a12cea7e28" gracePeriod=30
Sep 18 20:51:19.279754 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:19.279591    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kube-rbac-proxy" containerID="cri-o://5502264904c27b1357d30dc9cf742027c6664bf12677639bf937de9ed5aa4434" gracePeriod=30
Sep 18 20:51:20.106926 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:20.106888    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerID="5502264904c27b1357d30dc9cf742027c6664bf12677639bf937de9ed5aa4434" exitCode=2
Sep 18 20:51:20.107113 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:20.106949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" event={"ID":"a5558883-e8ba-4c7f-ad25-d38256e2e31a","Type":"ContainerDied","Data":"5502264904c27b1357d30dc9cf742027c6664bf12677639bf937de9ed5aa4434"}
Sep 18 20:51:21.750739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:21.750701    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 18 20:51:21.754487 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:21.754465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 18 20:51:24.127115 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.127084    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerID="c9bdd8eb18e6ddfefbb724db58e17792d60edaec03308711e98001a12cea7e28" exitCode=0
Sep 18 20:51:24.127448 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.127128    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" event={"ID":"a5558883-e8ba-4c7f-ad25-d38256e2e31a","Type":"ContainerDied","Data":"c9bdd8eb18e6ddfefbb724db58e17792d60edaec03308711e98001a12cea7e28"}
Sep 18 20:51:24.223807 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.223784    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:51:24.326248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.326170    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-rbac-proxy-sar-config\") pod \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") "
Sep 18 20:51:24.326248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.326209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kserve-provision-location\") pod \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") "
Sep 18 20:51:24.326248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.326230    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5558883-e8ba-4c7f-ad25-d38256e2e31a-proxy-tls\") pod \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") "
Sep 18 20:51:24.326620 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.326284    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6j9r\" (UniqueName: \"kubernetes.io/projected/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-api-access-s6j9r\") pod \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") "
Sep 18 20:51:24.326620 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.326336    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-cabundle-cert\") pod \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\" (UID: \"a5558883-e8ba-4c7f-ad25-d38256e2e31a\") "
Sep 18 20:51:24.326620 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.326546    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5558883-e8ba-4c7f-ad25-d38256e2e31a" (UID: "a5558883-e8ba-4c7f-ad25-d38256e2e31a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:24.326806 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.326688    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5558883-e8ba-4c7f-ad25-d38256e2e31a" (UID: "a5558883-e8ba-4c7f-ad25-d38256e2e31a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:51:24.326806 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.326767    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5558883-e8ba-4c7f-ad25-d38256e2e31a" (UID: "a5558883-e8ba-4c7f-ad25-d38256e2e31a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:51:24.328414 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.328389    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5558883-e8ba-4c7f-ad25-d38256e2e31a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5558883-e8ba-4c7f-ad25-d38256e2e31a" (UID: "a5558883-e8ba-4c7f-ad25-d38256e2e31a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:51:24.328503 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.328445    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-api-access-s6j9r" (OuterVolumeSpecName: "kube-api-access-s6j9r") pod "a5558883-e8ba-4c7f-ad25-d38256e2e31a" (UID: "a5558883-e8ba-4c7f-ad25-d38256e2e31a"). InnerVolumeSpecName "kube-api-access-s6j9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:51:24.427777 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.427742    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:24.427777 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.427776    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:24.428035 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.427790    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5558883-e8ba-4c7f-ad25-d38256e2e31a-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:24.428035 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.427800    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6j9r\" (UniqueName: \"kubernetes.io/projected/a5558883-e8ba-4c7f-ad25-d38256e2e31a-kube-api-access-s6j9r\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:24.428035 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.427808    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5558883-e8ba-4c7f-ad25-d38256e2e31a-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:24.913747 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:24.913711    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:51:25.135287 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:25.135250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf" event={"ID":"a5558883-e8ba-4c7f-ad25-d38256e2e31a","Type":"ContainerDied","Data":"8f6a1da7603471879082600c446b2d62301369c6f22daedfdf2b6e08cbe1dda4"}
Sep 18 20:51:25.135739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:25.135295    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"
Sep 18 20:51:25.135739 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:25.135330    2566 scope.go:117] "RemoveContainer" containerID="5502264904c27b1357d30dc9cf742027c6664bf12677639bf937de9ed5aa4434"
Sep 18 20:51:25.143543 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:25.143525    2566 scope.go:117] "RemoveContainer" containerID="c9bdd8eb18e6ddfefbb724db58e17792d60edaec03308711e98001a12cea7e28"
Sep 18 20:51:25.150738 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:25.150722    2566 scope.go:117] "RemoveContainer" containerID="a18da3a0c5dcfcd72022101af16a3e7904939ae4b054e92aba319f748f6a3c62"
Sep 18 20:51:25.157405 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:25.157382    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"]
Sep 18 20:51:25.163502 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:25.163484    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcf549695-hdpzf"]
Sep 18 20:51:25.461320 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:25.461277    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" path="/var/lib/kubelet/pods/a5558883-e8ba-4c7f-ad25-d38256e2e31a/volumes"
Sep 18 20:51:34.913384 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:34.913346    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:51:44.913204 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:44.913170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:51:53.108504 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.108472    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"]
Sep 18 20:51:53.108915 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.108707    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" containerID="cri-o://4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29" gracePeriod=30
Sep 18 20:51:53.108915 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.108746    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kube-rbac-proxy" containerID="cri-o://4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430" gracePeriod=30
Sep 18 20:51:53.157008 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.156979    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"]
Sep 18 20:51:53.157274 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.157263    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kube-rbac-proxy"
Sep 18 20:51:53.157317 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.157277    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kube-rbac-proxy"
Sep 18 20:51:53.157317 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.157297    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="storage-initializer"
Sep 18 20:51:53.157317 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.157302    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="storage-initializer"
Sep 18 20:51:53.157317 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.157308    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container"
Sep 18 20:51:53.157317 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.157313    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container"
Sep 18 20:51:53.157472 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.157375    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kube-rbac-proxy"
Sep 18 20:51:53.157472 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.157383    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5558883-e8ba-4c7f-ad25-d38256e2e31a" containerName="kserve-container"
Sep 18 20:51:53.160480 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.160466    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.163069 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.163051    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 20:51:53.170719 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.170696    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"]
Sep 18 20:51:53.239996 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.239968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.240125 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.240006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45h4p\" (UniqueName: \"kubernetes.io/projected/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-api-access-45h4p\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.240125 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.240063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.240125 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.240108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85943cc7-d186-4cab-89fd-8a584d1504b4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.240255 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.240142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85943cc7-d186-4cab-89fd-8a584d1504b4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.250992 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.250966    2566 generic.go:356] "Generic (PLEG): container finished" podID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerID="4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430" exitCode=2
Sep 18 20:51:53.251107 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.251046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" event={"ID":"2a3b5146-78b4-45e7-a113-d5ef12558d43","Type":"ContainerDied","Data":"4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430"}
Sep 18 20:51:53.340561 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.340529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85943cc7-d186-4cab-89fd-8a584d1504b4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.340694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.340570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85943cc7-d186-4cab-89fd-8a584d1504b4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.340694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.340589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.340694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.340618    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45h4p\" (UniqueName: \"kubernetes.io/projected/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-api-access-45h4p\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.340694 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.340649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.340929 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.340909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85943cc7-d186-4cab-89fd-8a584d1504b4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.341410 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.341390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.341474 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.341390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.342848 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.342832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85943cc7-d186-4cab-89fd-8a584d1504b4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.349624 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.349604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45h4p\" (UniqueName: \"kubernetes.io/projected/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-api-access-45h4p\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.470197 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.470119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:53.743842 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:53.743818    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"]
Sep 18 20:51:53.746186 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:51:53.746157    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85943cc7_d186_4cab_89fd_8a584d1504b4.slice/crio-4c46af3544bd308e5d6ce08dcd138542d2f459243dea98437c074a8c8ae39631 WatchSource:0}: Error finding container 4c46af3544bd308e5d6ce08dcd138542d2f459243dea98437c074a8c8ae39631: Status 404 returned error can't find the container with id 4c46af3544bd308e5d6ce08dcd138542d2f459243dea98437c074a8c8ae39631
Sep 18 20:51:54.256980 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:54.256942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" event={"ID":"85943cc7-d186-4cab-89fd-8a584d1504b4","Type":"ContainerStarted","Data":"4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970"}
Sep 18 20:51:54.257376 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:54.257001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" event={"ID":"85943cc7-d186-4cab-89fd-8a584d1504b4","Type":"ContainerStarted","Data":"4c46af3544bd308e5d6ce08dcd138542d2f459243dea98437c074a8c8ae39631"}
Sep 18 20:51:54.909067 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:54.909008    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 18 20:51:54.913377 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:54.913354    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 18 20:51:57.803770 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.803748    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:51:57.873853 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.873791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-rbac-proxy-sar-config\") pod \"2a3b5146-78b4-45e7-a113-d5ef12558d43\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") "
Sep 18 20:51:57.873853 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.873832    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3b5146-78b4-45e7-a113-d5ef12558d43-proxy-tls\") pod \"2a3b5146-78b4-45e7-a113-d5ef12558d43\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") "
Sep 18 20:51:57.874032 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.873862    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-cabundle-cert\") pod \"2a3b5146-78b4-45e7-a113-d5ef12558d43\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") "
Sep 18 20:51:57.874032 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.873974    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3b5146-78b4-45e7-a113-d5ef12558d43-kserve-provision-location\") pod \"2a3b5146-78b4-45e7-a113-d5ef12558d43\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") "
Sep 18 20:51:57.874122 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.874067    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjj8m\" (UniqueName: \"kubernetes.io/projected/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-api-access-bjj8m\") pod \"2a3b5146-78b4-45e7-a113-d5ef12558d43\" (UID: \"2a3b5146-78b4-45e7-a113-d5ef12558d43\") "
Sep 18 20:51:57.874247 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.874217    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a3b5146-78b4-45e7-a113-d5ef12558d43" (UID: "2a3b5146-78b4-45e7-a113-d5ef12558d43"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:51:57.874323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.874228    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a3b5146-78b4-45e7-a113-d5ef12558d43" (UID: "2a3b5146-78b4-45e7-a113-d5ef12558d43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:51:57.874323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.874271    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3b5146-78b4-45e7-a113-d5ef12558d43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a3b5146-78b4-45e7-a113-d5ef12558d43" (UID: "2a3b5146-78b4-45e7-a113-d5ef12558d43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:57.874323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.874300    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:57.874323 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.874316    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:57.875903 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.875878    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3b5146-78b4-45e7-a113-d5ef12558d43-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a3b5146-78b4-45e7-a113-d5ef12558d43" (UID: "2a3b5146-78b4-45e7-a113-d5ef12558d43"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:51:57.876000 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.875965    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-api-access-bjj8m" (OuterVolumeSpecName: "kube-api-access-bjj8m") pod "2a3b5146-78b4-45e7-a113-d5ef12558d43" (UID: "2a3b5146-78b4-45e7-a113-d5ef12558d43"). InnerVolumeSpecName "kube-api-access-bjj8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:51:57.975080 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.975055    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3b5146-78b4-45e7-a113-d5ef12558d43-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:57.975080 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.975076    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjj8m\" (UniqueName: \"kubernetes.io/projected/2a3b5146-78b4-45e7-a113-d5ef12558d43-kube-api-access-bjj8m\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:57.975209 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:57.975086    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3b5146-78b4-45e7-a113-d5ef12558d43-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:51:58.274734 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.274702    2566 generic.go:356] "Generic (PLEG): container finished" podID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerID="4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29" exitCode=0
Sep 18 20:51:58.274882 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.274756    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"
Sep 18 20:51:58.274882 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.274778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" event={"ID":"2a3b5146-78b4-45e7-a113-d5ef12558d43","Type":"ContainerDied","Data":"4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29"}
Sep 18 20:51:58.274882 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.274832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp" event={"ID":"2a3b5146-78b4-45e7-a113-d5ef12558d43","Type":"ContainerDied","Data":"5325142d49f8e483b5854c23cfe88dd06e5c32a24b3fcff9abaff1635938889a"}
Sep 18 20:51:58.274882 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.274857    2566 scope.go:117] "RemoveContainer" containerID="4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430"
Sep 18 20:51:58.276505 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.276485    2566 generic.go:356] "Generic (PLEG): container finished" podID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerID="4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970" exitCode=0
Sep 18 20:51:58.276643 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.276508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" event={"ID":"85943cc7-d186-4cab-89fd-8a584d1504b4","Type":"ContainerDied","Data":"4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970"}
Sep 18 20:51:58.283791 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.283679    2566 scope.go:117] "RemoveContainer" containerID="4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29"
Sep 18 20:51:58.290982 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.290966    2566 scope.go:117] "RemoveContainer" containerID="49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4"
Sep 18 20:51:58.299615 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.299597    2566 scope.go:117] "RemoveContainer" containerID="4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430"
Sep 18 20:51:58.299898 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:51:58.299879    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430\": container with ID starting with 4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430 not found: ID does not exist" containerID="4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430"
Sep 18 20:51:58.299971 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.299907    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430"} err="failed to get container status \"4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430\": rpc error: code = NotFound desc = could not find container \"4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430\": container with ID starting with 4d8710fa3a4b9bded24d8c676e01662cef8c269ccc2d318060dbdde135328430 not found: ID does not exist"
Sep 18 20:51:58.299971 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.299923    2566 scope.go:117] "RemoveContainer" containerID="4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29"
Sep 18 20:51:58.300213 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:51:58.300195    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29\": container with ID starting with 4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29 not found: ID does not exist" containerID="4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29"
Sep 18 20:51:58.300257 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.300218    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29"} err="failed to get container status \"4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29\": rpc error: code = NotFound desc = could not find container \"4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29\": container with ID starting with 4fa1a56e27649e15a1397bdf49d265d11c0af9202bbe379578f70c1385fe4a29 not found: ID does not exist"
Sep 18 20:51:58.300257 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.300231    2566 scope.go:117] "RemoveContainer" containerID="49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4"
Sep 18 20:51:58.300459 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:51:58.300439    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4\": container with ID starting with 49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4 not found: ID does not exist" containerID="49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4"
Sep 18 20:51:58.300519 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.300467    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4"} err="failed to get container status \"49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4\": rpc error: code = NotFound desc = could not find container \"49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4\": container with ID starting with 49f89f4213fa0f80012395ce1380b656f9b8dfce648a8c784fed0aa8bbe908a4 not found: ID does not exist"
Sep 18 20:51:58.319344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.319315    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"]
Sep 18 20:51:58.320642 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:58.320622    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-m5krp"]
Sep 18 20:51:59.283639 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:59.283602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" event={"ID":"85943cc7-d186-4cab-89fd-8a584d1504b4","Type":"ContainerStarted","Data":"45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738"}
Sep 18 20:51:59.284005 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:59.283659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" event={"ID":"85943cc7-d186-4cab-89fd-8a584d1504b4","Type":"ContainerStarted","Data":"b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8"}
Sep 18 20:51:59.284005 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:59.283855    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:51:59.303387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:59.303340    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podStartSLOduration=2.394899418 podStartE2EDuration="6.303324712s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.908425294s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:51:59.300944139 +0000 UTC m=+1888.331870292" watchObservedRunningTime="2026-09-18 20:51:59.303324712 +0000 UTC m=+1888.334250896"
Sep 18 20:51:59.459249 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:51:59.459214    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" path="/var/lib/kubelet/pods/2a3b5146-78b4-45e7-a113-d5ef12558d43/volumes"
Sep 18 20:52:00.287816 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:00.287786    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:52:00.289164 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:00.289138    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:52:01.291966 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:01.291931    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:52:06.297798 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:06.297767    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:52:06.298437 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:06.298409    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:52:10.729120 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.729085    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"]
Sep 18 20:52:10.729471 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.729404    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kube-rbac-proxy"
Sep 18 20:52:10.729471 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.729417    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kube-rbac-proxy"
Sep 18 20:52:10.729471 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.729437    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container"
Sep 18 20:52:10.729471 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.729443    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container"
Sep 18 20:52:10.729471 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.729466    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="storage-initializer"
Sep 18 20:52:10.729471 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.729472    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="storage-initializer"
Sep 18 20:52:10.729657 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.729523    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kube-rbac-proxy"
Sep 18 20:52:10.729657 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.729532    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3b5146-78b4-45e7-a113-d5ef12558d43" containerName="kserve-container"
Sep 18 20:52:10.732641 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.732618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.735082 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.735060    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:52:10.735222 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.735058    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 18 20:52:10.744144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.744120    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"]
Sep 18 20:52:10.773354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.773322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.773481 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.773376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8539d8a1-0044-43d0-9b09-d68b87b61b16-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.773481 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.773406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gmb\" (UniqueName: \"kubernetes.io/projected/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-api-access-p2gmb\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.773588 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.773517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.773639 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.773589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8539d8a1-0044-43d0-9b09-d68b87b61b16-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.874830 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.874798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.874976 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.874852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8539d8a1-0044-43d0-9b09-d68b87b61b16-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.874976 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.874895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.874976 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.874931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8539d8a1-0044-43d0-9b09-d68b87b61b16-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.875186 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.875057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gmb\" (UniqueName: \"kubernetes.io/projected/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-api-access-p2gmb\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.875186 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:52:10.875063    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 18 20:52:10.875287 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:52:10.875211    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8539d8a1-0044-43d0-9b09-d68b87b61b16-proxy-tls podName:8539d8a1-0044-43d0-9b09-d68b87b61b16 nodeName:}" failed. No retries permitted until 2026-09-18 20:52:11.375186913 +0000 UTC m=+1900.406113048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8539d8a1-0044-43d0-9b09-d68b87b61b16-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" (UID: "8539d8a1-0044-43d0-9b09-d68b87b61b16") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 18 20:52:10.875398 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.875376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8539d8a1-0044-43d0-9b09-d68b87b61b16-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.875656 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.875636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.875693 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.875636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:10.883625 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:10.883599    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gmb\" (UniqueName: \"kubernetes.io/projected/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-api-access-p2gmb\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:11.379555 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:11.379521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8539d8a1-0044-43d0-9b09-d68b87b61b16-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:11.381843 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:11.381819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8539d8a1-0044-43d0-9b09-d68b87b61b16-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:11.643161 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:11.643083    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:11.915889 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:11.915858    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"]
Sep 18 20:52:11.918342 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:52:11.918313    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8539d8a1_0044_43d0_9b09_d68b87b61b16.slice/crio-3d87171c4ffc780391c35b8d7d163884016111ffce26321031fddf1d9786b498 WatchSource:0}: Error finding container 3d87171c4ffc780391c35b8d7d163884016111ffce26321031fddf1d9786b498: Status 404 returned error can't find the container with id 3d87171c4ffc780391c35b8d7d163884016111ffce26321031fddf1d9786b498
Sep 18 20:52:12.340761 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:12.340726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" event={"ID":"8539d8a1-0044-43d0-9b09-d68b87b61b16","Type":"ContainerStarted","Data":"c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d"}
Sep 18 20:52:12.340957 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:12.340813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" event={"ID":"8539d8a1-0044-43d0-9b09-d68b87b61b16","Type":"ContainerStarted","Data":"3d87171c4ffc780391c35b8d7d163884016111ffce26321031fddf1d9786b498"}
Sep 18 20:52:16.298545 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:16.298509    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:52:16.357640 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:16.357612    2566 generic.go:356] "Generic (PLEG): container finished" podID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerID="c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d" exitCode=0
Sep 18 20:52:16.357746 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:16.357657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" event={"ID":"8539d8a1-0044-43d0-9b09-d68b87b61b16","Type":"ContainerDied","Data":"c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d"}
Sep 18 20:52:16.358935 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:16.358919    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:52:20.378954 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:20.378914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" event={"ID":"8539d8a1-0044-43d0-9b09-d68b87b61b16","Type":"ContainerStarted","Data":"6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd"}
Sep 18 20:52:20.379354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:20.378985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" event={"ID":"8539d8a1-0044-43d0-9b09-d68b87b61b16","Type":"ContainerStarted","Data":"97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f"}
Sep 18 20:52:20.379354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:20.379273    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:20.379354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:20.379331    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:20.380785 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:20.380757    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:52:20.399745 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:20.399704    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podStartSLOduration=2.532230423 podStartE2EDuration="10.399689623s" totalImagesPullingTime="3.552375248s" totalInitContainerRuntime="4.315083952s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:52:20.399240514 +0000 UTC m=+1909.430166654" watchObservedRunningTime="2026-09-18 20:52:20.399689623 +0000 UTC m=+1909.430615762"
Sep 18 20:52:21.383295 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:21.383256    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:52:26.298680 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:26.298641    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:52:26.389009 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:26.388977    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:26.389566 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:26.389535    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 18 20:52:36.298859 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:36.298818    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:52:36.390584 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:36.390558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:52:46.299030 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:46.298978    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:52:51.694615 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:51.694582    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"]
Sep 18 20:52:51.695131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:51.694838    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kserve-container" containerID="cri-o://97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f" gracePeriod=30
Sep 18 20:52:51.695131 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:51.694880    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy" containerID="cri-o://6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd" gracePeriod=30
Sep 18 20:52:52.505937 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:52.505899    2566 generic.go:356] "Generic (PLEG): container finished" podID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerID="6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd" exitCode=2
Sep 18 20:52:52.506123 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:52.505970    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" event={"ID":"8539d8a1-0044-43d0-9b09-d68b87b61b16","Type":"ContainerDied","Data":"6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd"}
Sep 18 20:52:56.298439 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:56.298402    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:52:56.383670 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:52:56.383629    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 18 20:53:01.383811 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:01.383766    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 18 20:53:06.298945 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:06.298907    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:53:06.384295 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:06.384257    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 18 20:53:06.384449 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:06.384363    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:53:11.384506 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:11.384467    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 18 20:53:11.456710 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:11.456670    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 18 20:53:16.384167 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:16.384125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 18 20:53:21.383871 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:21.383833    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 18 20:53:21.459178 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:21.459153    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:53:22.328472 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.328449    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:53:22.449704 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.449677    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8539d8a1-0044-43d0-9b09-d68b87b61b16-proxy-tls\") pod \"8539d8a1-0044-43d0-9b09-d68b87b61b16\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") "
Sep 18 20:53:22.450153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.449726    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-rbac-proxy-sar-config\") pod \"8539d8a1-0044-43d0-9b09-d68b87b61b16\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") "
Sep 18 20:53:22.450153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.449755    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2gmb\" (UniqueName: \"kubernetes.io/projected/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-api-access-p2gmb\") pod \"8539d8a1-0044-43d0-9b09-d68b87b61b16\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") "
Sep 18 20:53:22.450153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.449789    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8539d8a1-0044-43d0-9b09-d68b87b61b16-kserve-provision-location\") pod \"8539d8a1-0044-43d0-9b09-d68b87b61b16\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") "
Sep 18 20:53:22.450153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.449822    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-cabundle-cert\") pod \"8539d8a1-0044-43d0-9b09-d68b87b61b16\" (UID: \"8539d8a1-0044-43d0-9b09-d68b87b61b16\") "
Sep 18 20:53:22.450396 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.450170    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8539d8a1-0044-43d0-9b09-d68b87b61b16" (UID: "8539d8a1-0044-43d0-9b09-d68b87b61b16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:53:22.450452 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.450399    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8539d8a1-0044-43d0-9b09-d68b87b61b16" (UID: "8539d8a1-0044-43d0-9b09-d68b87b61b16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:53:22.452036 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.451989    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8539d8a1-0044-43d0-9b09-d68b87b61b16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8539d8a1-0044-43d0-9b09-d68b87b61b16" (UID: "8539d8a1-0044-43d0-9b09-d68b87b61b16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:53:22.452036 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.452024    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-api-access-p2gmb" (OuterVolumeSpecName: "kube-api-access-p2gmb") pod "8539d8a1-0044-43d0-9b09-d68b87b61b16" (UID: "8539d8a1-0044-43d0-9b09-d68b87b61b16"). InnerVolumeSpecName "kube-api-access-p2gmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:53:22.460857 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.460802    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8539d8a1-0044-43d0-9b09-d68b87b61b16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8539d8a1-0044-43d0-9b09-d68b87b61b16" (UID: "8539d8a1-0044-43d0-9b09-d68b87b61b16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:53:22.551153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.551121    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8539d8a1-0044-43d0-9b09-d68b87b61b16-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:22.551153 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.551156    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:22.551374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.551171    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gmb\" (UniqueName: \"kubernetes.io/projected/8539d8a1-0044-43d0-9b09-d68b87b61b16-kube-api-access-p2gmb\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:22.551374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.551185    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8539d8a1-0044-43d0-9b09-d68b87b61b16-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:22.551374 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.551199    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8539d8a1-0044-43d0-9b09-d68b87b61b16-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:22.620330 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.620302    2566 generic.go:356] "Generic (PLEG): container finished" podID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerID="97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f" exitCode=137
Sep 18 20:53:22.620449 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.620369    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"
Sep 18 20:53:22.620449 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.620378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" event={"ID":"8539d8a1-0044-43d0-9b09-d68b87b61b16","Type":"ContainerDied","Data":"97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f"}
Sep 18 20:53:22.620449 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.620427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk" event={"ID":"8539d8a1-0044-43d0-9b09-d68b87b61b16","Type":"ContainerDied","Data":"3d87171c4ffc780391c35b8d7d163884016111ffce26321031fddf1d9786b498"}
Sep 18 20:53:22.620567 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.620451    2566 scope.go:117] "RemoveContainer" containerID="6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd"
Sep 18 20:53:22.629132 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.629111    2566 scope.go:117] "RemoveContainer" containerID="97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f"
Sep 18 20:53:22.636197 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.636181    2566 scope.go:117] "RemoveContainer" containerID="c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d"
Sep 18 20:53:22.643318 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.643299    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"]
Sep 18 20:53:22.643580 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.643567    2566 scope.go:117] "RemoveContainer" containerID="6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd"
Sep 18 20:53:22.643830 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:53:22.643812    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd\": container with ID starting with 6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd not found: ID does not exist" containerID="6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd"
Sep 18 20:53:22.643869 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.643839    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd"} err="failed to get container status \"6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd\": rpc error: code = NotFound desc = could not find container \"6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd\": container with ID starting with 6e62d7f1f4d2821b6eb9bf23033a9056a4a00eedef884974a348cfb8827bfebd not found: ID does not exist"
Sep 18 20:53:22.643869 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.643867    2566 scope.go:117] "RemoveContainer" containerID="97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f"
Sep 18 20:53:22.644138 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:53:22.644123    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f\": container with ID starting with 97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f not found: ID does not exist" containerID="97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f"
Sep 18 20:53:22.644183 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.644142    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f"} err="failed to get container status \"97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f\": rpc error: code = NotFound desc = could not find container \"97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f\": container with ID starting with 97d0985010724d640258becc43713055a25a64db23444c50023328cd09a1164f not found: ID does not exist"
Sep 18 20:53:22.644183 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.644154    2566 scope.go:117] "RemoveContainer" containerID="c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d"
Sep 18 20:53:22.644393 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:53:22.644377    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d\": container with ID starting with c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d not found: ID does not exist" containerID="c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d"
Sep 18 20:53:22.644433 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.644397    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d"} err="failed to get container status \"c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d\": rpc error: code = NotFound desc = could not find container \"c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d\": container with ID starting with c22f309ae5762e304b0a4b17522ab330526a0b77ab624c6b5719d6227ba1c33d not found: ID does not exist"
Sep 18 20:53:22.647588 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:22.647569    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-s4qrk"]
Sep 18 20:53:23.280970 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:23.280937    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"]
Sep 18 20:53:23.281256 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:23.281227    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container" containerID="cri-o://b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8" gracePeriod=30
Sep 18 20:53:23.281318 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:23.281264    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kube-rbac-proxy" containerID="cri-o://45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738" gracePeriod=30
Sep 18 20:53:23.459911 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:23.459882    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" path="/var/lib/kubelet/pods/8539d8a1-0044-43d0-9b09-d68b87b61b16/volumes"
Sep 18 20:53:23.627227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:23.627144    2566 generic.go:356] "Generic (PLEG): container finished" podID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerID="45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738" exitCode=2
Sep 18 20:53:23.627227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:23.627195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" event={"ID":"85943cc7-d186-4cab-89fd-8a584d1504b4","Type":"ContainerDied","Data":"45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738"}
Sep 18 20:53:26.292366 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:26.292328    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 18 20:53:27.828787 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.828766    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:53:27.894305 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.894276    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-cabundle-cert\") pod \"85943cc7-d186-4cab-89fd-8a584d1504b4\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") "
Sep 18 20:53:27.894475 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.894314    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45h4p\" (UniqueName: \"kubernetes.io/projected/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-api-access-45h4p\") pod \"85943cc7-d186-4cab-89fd-8a584d1504b4\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") "
Sep 18 20:53:27.894475 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.894366    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-rbac-proxy-sar-config\") pod \"85943cc7-d186-4cab-89fd-8a584d1504b4\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") "
Sep 18 20:53:27.894475 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.894387    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85943cc7-d186-4cab-89fd-8a584d1504b4-proxy-tls\") pod \"85943cc7-d186-4cab-89fd-8a584d1504b4\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") "
Sep 18 20:53:27.894475 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.894438    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85943cc7-d186-4cab-89fd-8a584d1504b4-kserve-provision-location\") pod \"85943cc7-d186-4cab-89fd-8a584d1504b4\" (UID: \"85943cc7-d186-4cab-89fd-8a584d1504b4\") "
Sep 18 20:53:27.894712 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.894684    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85943cc7-d186-4cab-89fd-8a584d1504b4" (UID: "85943cc7-d186-4cab-89fd-8a584d1504b4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:53:27.894822 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.894748    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85943cc7-d186-4cab-89fd-8a584d1504b4" (UID: "85943cc7-d186-4cab-89fd-8a584d1504b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:53:27.894822 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.894774    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85943cc7-d186-4cab-89fd-8a584d1504b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85943cc7-d186-4cab-89fd-8a584d1504b4" (UID: "85943cc7-d186-4cab-89fd-8a584d1504b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:53:27.896487 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.896469    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85943cc7-d186-4cab-89fd-8a584d1504b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85943cc7-d186-4cab-89fd-8a584d1504b4" (UID: "85943cc7-d186-4cab-89fd-8a584d1504b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:53:27.896487 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.896478    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-api-access-45h4p" (OuterVolumeSpecName: "kube-api-access-45h4p") pod "85943cc7-d186-4cab-89fd-8a584d1504b4" (UID: "85943cc7-d186-4cab-89fd-8a584d1504b4"). InnerVolumeSpecName "kube-api-access-45h4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:53:27.995644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.995579    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:27.995644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.995602    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45h4p\" (UniqueName: \"kubernetes.io/projected/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-api-access-45h4p\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:27.995644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.995611    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85943cc7-d186-4cab-89fd-8a584d1504b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:27.995644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.995621    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85943cc7-d186-4cab-89fd-8a584d1504b4-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:27.995644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:27.995630    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85943cc7-d186-4cab-89fd-8a584d1504b4-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:53:28.486816 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.486793    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"]
Sep 18 20:53:28.487094 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487081    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy"
Sep 18 20:53:28.487141 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487096    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy"
Sep 18 20:53:28.487141 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487104    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="storage-initializer"
Sep 18 20:53:28.487141 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487110    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="storage-initializer"
Sep 18 20:53:28.487141 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487120    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kube-rbac-proxy"
Sep 18 20:53:28.487141 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487126    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kube-rbac-proxy"
Sep 18 20:53:28.487141 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487133    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="storage-initializer"
Sep 18 20:53:28.487141 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487138    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="storage-initializer"
Sep 18 20:53:28.487354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487144    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kserve-container"
Sep 18 20:53:28.487354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487149    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kserve-container"
Sep 18 20:53:28.487354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487158    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container"
Sep 18 20:53:28.487354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487163    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container"
Sep 18 20:53:28.487354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487221    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kserve-container"
Sep 18 20:53:28.487354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487228    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8539d8a1-0044-43d0-9b09-d68b87b61b16" containerName="kube-rbac-proxy"
Sep 18 20:53:28.487354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487233    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kserve-container"
Sep 18 20:53:28.487354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.487239    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerName="kube-rbac-proxy"
Sep 18 20:53:28.490409 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.490392    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.493216 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.493192    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 18 20:53:28.493331 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.493217    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 20:53:28.504768 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.504628    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"]
Sep 18 20:53:28.524792 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.524769    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"]
Sep 18 20:53:28.527924 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.527906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.530144 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.530126    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 20:53:28.540843 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.540819    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"]
Sep 18 20:53:28.600942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.600916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.601082 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.600948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb87e6d-3283-469d-9e30-ba78841c8ada-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.601082 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.600964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c1f8293-a028-472c-828b-64ebbf620ffe-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.601082 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.601000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.601082 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.601036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ghw\" (UniqueName: \"kubernetes.io/projected/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-api-access-c7ghw\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.601082 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.601053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb87e6d-3283-469d-9e30-ba78841c8ada-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.601082 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.601067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.601297 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.601097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c1f8293-a028-472c-828b-64ebbf620ffe-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.601297 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.601153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.601297 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.601203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hht\" (UniqueName: \"kubernetes.io/projected/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-api-access-q7hht\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.649100 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.649070    2566 generic.go:356] "Generic (PLEG): container finished" podID="85943cc7-d186-4cab-89fd-8a584d1504b4" containerID="b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8" exitCode=0
Sep 18 20:53:28.649242 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.649103    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" event={"ID":"85943cc7-d186-4cab-89fd-8a584d1504b4","Type":"ContainerDied","Data":"b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8"}
Sep 18 20:53:28.649242 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.649149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d" event={"ID":"85943cc7-d186-4cab-89fd-8a584d1504b4","Type":"ContainerDied","Data":"4c46af3544bd308e5d6ce08dcd138542d2f459243dea98437c074a8c8ae39631"}
Sep 18 20:53:28.649242 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.649142    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"
Sep 18 20:53:28.649242 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.649174    2566 scope.go:117] "RemoveContainer" containerID="45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738"
Sep 18 20:53:28.657292 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.657096    2566 scope.go:117] "RemoveContainer" containerID="b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8"
Sep 18 20:53:28.664571 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.664552    2566 scope.go:117] "RemoveContainer" containerID="4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970"
Sep 18 20:53:28.675339 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.675313    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"]
Sep 18 20:53:28.675667 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.675650    2566 scope.go:117] "RemoveContainer" containerID="45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738"
Sep 18 20:53:28.675940 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:53:28.675921    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738\": container with ID starting with 45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738 not found: ID does not exist" containerID="45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738"
Sep 18 20:53:28.675986 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.675948    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738"} err="failed to get container status \"45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738\": rpc error: code = NotFound desc = could not find container \"45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738\": container with ID starting with 45a767a39b3cc636ebdaed25f613f1a4b23c4abb5e5a644d8b721087d797c738 not found: ID does not exist"
Sep 18 20:53:28.675986 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.675967    2566 scope.go:117] "RemoveContainer" containerID="b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8"
Sep 18 20:53:28.676361 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:53:28.676337    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8\": container with ID starting with b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8 not found: ID does not exist" containerID="b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8"
Sep 18 20:53:28.676448 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.676367    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8"} err="failed to get container status \"b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8\": rpc error: code = NotFound desc = could not find container \"b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8\": container with ID starting with b02648ce4089d6bee1e7431753b67b6f1b39b02150c7c333eb12ac788dd743f8 not found: ID does not exist"
Sep 18 20:53:28.676448 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.676388    2566 scope.go:117] "RemoveContainer" containerID="4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970"
Sep 18 20:53:28.676839 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:53:28.676773    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970\": container with ID starting with 4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970 not found: ID does not exist" containerID="4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970"
Sep 18 20:53:28.676839 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.676805    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970"} err="failed to get container status \"4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970\": rpc error: code = NotFound desc = could not find container \"4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970\": container with ID starting with 4bc60efde29a6d1fb9e9315d6d8b3776b771890e15f4395d57c86bb4a06a3970 not found: ID does not exist"
Sep 18 20:53:28.678642 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.678623    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-9xg8d"]
Sep 18 20:53:28.702211 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.702314 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hht\" (UniqueName: \"kubernetes.io/projected/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-api-access-q7hht\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.702378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.702378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb87e6d-3283-469d-9e30-ba78841c8ada-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.702378 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702368    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c1f8293-a028-472c-828b-64ebbf620ffe-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.702541 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:53:28.702493    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 18 20:53:28.702598 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:53:28.702557    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1f8293-a028-472c-828b-64ebbf620ffe-proxy-tls podName:9c1f8293-a028-472c-828b-64ebbf620ffe nodeName:}" failed. No retries permitted until 2026-09-18 20:53:29.202534777 +0000 UTC m=+1978.233460915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9c1f8293-a028-472c-828b-64ebbf620ffe-proxy-tls") pod "isvc-canary-rollback-predictor-7f7b644c9-lptf7" (UID: "9c1f8293-a028-472c-828b-64ebbf620ffe") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 18 20:53:28.703058 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.703058 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ghw\" (UniqueName: \"kubernetes.io/projected/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-api-access-c7ghw\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.703058 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb87e6d-3283-469d-9e30-ba78841c8ada-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.703058 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.703058 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.703058 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.702975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c1f8293-a028-472c-828b-64ebbf620ffe-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.703058 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.703008    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.703476 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.703348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c1f8293-a028-472c-828b-64ebbf620ffe-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.703476 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.703467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb87e6d-3283-469d-9e30-ba78841c8ada-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.703592 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.703577    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.703803 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.703784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.704913 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.704893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb87e6d-3283-469d-9e30-ba78841c8ada-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.710782 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.710759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ghw\" (UniqueName: \"kubernetes.io/projected/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-api-access-c7ghw\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:28.710930 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.710916    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hht\" (UniqueName: \"kubernetes.io/projected/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-api-access-q7hht\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:28.805036 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:28.805001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:29.078068 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:29.078042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"]
Sep 18 20:53:29.080472 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:53:29.080451    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb87e6d_3283_469d_9e30_ba78841c8ada.slice/crio-dba0e591118379384d10518252a93cb7c8f30cfde96f5cd372b33c15dae4cccd WatchSource:0}: Error finding container dba0e591118379384d10518252a93cb7c8f30cfde96f5cd372b33c15dae4cccd: Status 404 returned error can't find the container with id dba0e591118379384d10518252a93cb7c8f30cfde96f5cd372b33c15dae4cccd
Sep 18 20:53:29.206538 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:29.206510    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c1f8293-a028-472c-828b-64ebbf620ffe-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:29.208735 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:29.208707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c1f8293-a028-472c-828b-64ebbf620ffe-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-lptf7\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:29.440076 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:29.439975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:29.462034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:29.461990    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85943cc7-d186-4cab-89fd-8a584d1504b4" path="/var/lib/kubelet/pods/85943cc7-d186-4cab-89fd-8a584d1504b4/volumes"
Sep 18 20:53:29.655673 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:29.655632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" event={"ID":"efb87e6d-3283-469d-9e30-ba78841c8ada","Type":"ContainerStarted","Data":"51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61"}
Sep 18 20:53:29.655843 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:29.655709    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" event={"ID":"efb87e6d-3283-469d-9e30-ba78841c8ada","Type":"ContainerStarted","Data":"dba0e591118379384d10518252a93cb7c8f30cfde96f5cd372b33c15dae4cccd"}
Sep 18 20:53:29.713494 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:53:29.713451    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1f8293_a028_472c_828b_64ebbf620ffe.slice/crio-532adf5446c707eddc0ab16650508ce88b17aaac58c01d481ee4595552234a59 WatchSource:0}: Error finding container 532adf5446c707eddc0ab16650508ce88b17aaac58c01d481ee4595552234a59: Status 404 returned error can't find the container with id 532adf5446c707eddc0ab16650508ce88b17aaac58c01d481ee4595552234a59
Sep 18 20:53:29.713792 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:29.713769    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"]
Sep 18 20:53:30.662397 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:30.662362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" event={"ID":"9c1f8293-a028-472c-828b-64ebbf620ffe","Type":"ContainerStarted","Data":"5cc80a7cbfd11e9b46ed2d6d4ae8ab9910d39b59f3f24e5fbd5fc1b45e77bb86"}
Sep 18 20:53:30.662769 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:30.662431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" event={"ID":"9c1f8293-a028-472c-828b-64ebbf620ffe","Type":"ContainerStarted","Data":"532adf5446c707eddc0ab16650508ce88b17aaac58c01d481ee4595552234a59"}
Sep 18 20:53:33.676472 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:33.676439    2566 generic.go:356] "Generic (PLEG): container finished" podID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerID="51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61" exitCode=0
Sep 18 20:53:33.676849 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:33.676511    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" event={"ID":"efb87e6d-3283-469d-9e30-ba78841c8ada","Type":"ContainerDied","Data":"51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61"}
Sep 18 20:53:33.678135 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:33.678113    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerID="5cc80a7cbfd11e9b46ed2d6d4ae8ab9910d39b59f3f24e5fbd5fc1b45e77bb86" exitCode=0
Sep 18 20:53:33.678213 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:33.678154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" event={"ID":"9c1f8293-a028-472c-828b-64ebbf620ffe","Type":"ContainerDied","Data":"5cc80a7cbfd11e9b46ed2d6d4ae8ab9910d39b59f3f24e5fbd5fc1b45e77bb86"}
Sep 18 20:53:34.684028 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.683968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" event={"ID":"efb87e6d-3283-469d-9e30-ba78841c8ada","Type":"ContainerStarted","Data":"ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10"}
Sep 18 20:53:34.684491 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.684056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" event={"ID":"efb87e6d-3283-469d-9e30-ba78841c8ada","Type":"ContainerStarted","Data":"54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520"}
Sep 18 20:53:34.684491 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.684335    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:34.685987 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.685967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" event={"ID":"9c1f8293-a028-472c-828b-64ebbf620ffe","Type":"ContainerStarted","Data":"172f016cf82b7119e48388c5d116adcd8ccbac55deb27f774b7c33be31e42d24"}
Sep 18 20:53:34.686118 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.686024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" event={"ID":"9c1f8293-a028-472c-828b-64ebbf620ffe","Type":"ContainerStarted","Data":"e66641593e35045248fd3d4b6627a9df09bea8201ce61fb65cc9966bc4f480cd"}
Sep 18 20:53:34.686217 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.686197    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:34.686259 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.686249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:34.687618 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.687598    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:53:34.703628 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.703581    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podStartSLOduration=2.757012935 podStartE2EDuration="6.703568033s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.946555098s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:53:34.701968986 +0000 UTC m=+1983.732895126" watchObservedRunningTime="2026-09-18 20:53:34.703568033 +0000 UTC m=+1983.734494172"
Sep 18 20:53:34.718900 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:34.718861    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podStartSLOduration=3.045258246 podStartE2EDuration="6.718848585s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.673590339s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:53:34.717909443 +0000 UTC m=+1983.748835582" watchObservedRunningTime="2026-09-18 20:53:34.718848585 +0000 UTC m=+1983.749774725"
Sep 18 20:53:35.690517 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:35.690473    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:53:35.690908 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:35.690612    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:35.691749 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:35.691723    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:53:36.693755 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:36.693707    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:53:40.694686 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:40.694658    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:53:40.695174 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:40.695147    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:53:41.697904 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:41.697876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:53:41.698397 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:41.698371    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:53:50.697108 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:50.695715    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:53:51.698985 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:53:51.698946    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:54:00.695250 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:00.695211    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:54:01.699007 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:01.698966    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:54:10.695519 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:10.695478    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:54:11.698558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:11.698515    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:54:20.695325 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:20.695280    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:54:21.698847 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:21.698810    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:54:30.695966 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:30.695927    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:54:31.699034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:31.698973    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:54:40.695425 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:40.695384    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:54:41.698373 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:41.698337    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:54:50.696166 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:50.696131    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:54:51.699034 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:51.698989    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:54:58.549962 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:58.549880    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"]
Sep 18 20:54:58.550522 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:58.550221    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kube-rbac-proxy" containerID="cri-o://ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10" gracePeriod=30
Sep 18 20:54:58.550522 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:58.550201    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" containerID="cri-o://54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520" gracePeriod=30
Sep 18 20:54:59.026606 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:59.026578    2566 generic.go:356] "Generic (PLEG): container finished" podID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerID="ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10" exitCode=2
Sep 18 20:54:59.026765 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:54:59.026656    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" event={"ID":"efb87e6d-3283-469d-9e30-ba78841c8ada","Type":"ContainerDied","Data":"ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10"}
Sep 18 20:55:01.694462 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:01.694421    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 18 20:55:01.698737 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:01.698703    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 18 20:55:03.390942 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.390918    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:55:03.465657 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.465582    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-cabundle-cert\") pod \"efb87e6d-3283-469d-9e30-ba78841c8ada\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") "
Sep 18 20:55:03.465657 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.465620    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7ghw\" (UniqueName: \"kubernetes.io/projected/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-api-access-c7ghw\") pod \"efb87e6d-3283-469d-9e30-ba78841c8ada\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") "
Sep 18 20:55:03.465886 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.465665    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb87e6d-3283-469d-9e30-ba78841c8ada-kserve-provision-location\") pod \"efb87e6d-3283-469d-9e30-ba78841c8ada\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") "
Sep 18 20:55:03.465886 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.465704    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb87e6d-3283-469d-9e30-ba78841c8ada-proxy-tls\") pod \"efb87e6d-3283-469d-9e30-ba78841c8ada\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") "
Sep 18 20:55:03.465886 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.465746    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-rbac-proxy-sar-config\") pod \"efb87e6d-3283-469d-9e30-ba78841c8ada\" (UID: \"efb87e6d-3283-469d-9e30-ba78841c8ada\") "
Sep 18 20:55:03.466094 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.466067    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "efb87e6d-3283-469d-9e30-ba78841c8ada" (UID: "efb87e6d-3283-469d-9e30-ba78841c8ada"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:55:03.466094 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.466082    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb87e6d-3283-469d-9e30-ba78841c8ada-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efb87e6d-3283-469d-9e30-ba78841c8ada" (UID: "efb87e6d-3283-469d-9e30-ba78841c8ada"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:55:03.466184 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.466163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "efb87e6d-3283-469d-9e30-ba78841c8ada" (UID: "efb87e6d-3283-469d-9e30-ba78841c8ada"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:55:03.467905 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.467881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-api-access-c7ghw" (OuterVolumeSpecName: "kube-api-access-c7ghw") pod "efb87e6d-3283-469d-9e30-ba78841c8ada" (UID: "efb87e6d-3283-469d-9e30-ba78841c8ada"). InnerVolumeSpecName "kube-api-access-c7ghw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:55:03.468028 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.467954    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb87e6d-3283-469d-9e30-ba78841c8ada-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "efb87e6d-3283-469d-9e30-ba78841c8ada" (UID: "efb87e6d-3283-469d-9e30-ba78841c8ada"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:55:03.566208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.566179    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:55:03.566208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.566202    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7ghw\" (UniqueName: \"kubernetes.io/projected/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-api-access-c7ghw\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:55:03.566208 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.566212    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb87e6d-3283-469d-9e30-ba78841c8ada-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:55:03.566436 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.566221    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb87e6d-3283-469d-9e30-ba78841c8ada-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:55:03.566436 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:03.566230    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb87e6d-3283-469d-9e30-ba78841c8ada-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:55:04.049679 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.049646    2566 generic.go:356] "Generic (PLEG): container finished" podID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerID="54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520" exitCode=0
Sep 18 20:55:04.049927 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.049702    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"
Sep 18 20:55:04.049927 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.049712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" event={"ID":"efb87e6d-3283-469d-9e30-ba78841c8ada","Type":"ContainerDied","Data":"54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520"}
Sep 18 20:55:04.049927 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.049762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn" event={"ID":"efb87e6d-3283-469d-9e30-ba78841c8ada","Type":"ContainerDied","Data":"dba0e591118379384d10518252a93cb7c8f30cfde96f5cd372b33c15dae4cccd"}
Sep 18 20:55:04.049927 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.049786    2566 scope.go:117] "RemoveContainer" containerID="ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10"
Sep 18 20:55:04.058949 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.058778    2566 scope.go:117] "RemoveContainer" containerID="54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520"
Sep 18 20:55:04.068367 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.068190    2566 scope.go:117] "RemoveContainer" containerID="51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61"
Sep 18 20:55:04.075872 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.075853    2566 scope.go:117] "RemoveContainer" containerID="ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10"
Sep 18 20:55:04.076145 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:55:04.076125    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10\": container with ID starting with ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10 not found: ID does not exist" containerID="ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10"
Sep 18 20:55:04.076225 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.076151    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10"} err="failed to get container status \"ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10\": rpc error: code = NotFound desc = could not find container \"ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10\": container with ID starting with ba280b9062238163e1656e61de3e5421dd5c6d28af156215fdc652259365fe10 not found: ID does not exist"
Sep 18 20:55:04.076225 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.076168    2566 scope.go:117] "RemoveContainer" containerID="54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520"
Sep 18 20:55:04.076389 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:55:04.076374    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520\": container with ID starting with 54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520 not found: ID does not exist" containerID="54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520"
Sep 18 20:55:04.076448 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.076392    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520"} err="failed to get container status \"54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520\": rpc error: code = NotFound desc = could not find container \"54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520\": container with ID starting with 54a509670e0e57340c09688f4686cb86a231013db0e245b0d674c64dee9e8520 not found: ID does not exist"
Sep 18 20:55:04.076448 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.076405    2566 scope.go:117] "RemoveContainer" containerID="51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61"
Sep 18 20:55:04.076602 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:55:04.076587    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61\": container with ID starting with 51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61 not found: ID does not exist" containerID="51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61"
Sep 18 20:55:04.076668 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.076603    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61"} err="failed to get container status \"51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61\": rpc error: code = NotFound desc = could not find container \"51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61\": container with ID starting with 51506a663187c6423d08fb71256aae4ebe35f21ea37b34a73c9ec59737bb9c61 not found: ID does not exist"
Sep 18 20:55:04.077587 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.077565    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"]
Sep 18 20:55:04.081505 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:04.081485    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-7rjsn"]
Sep 18 20:55:05.459825 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:05.459791    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" path="/var/lib/kubelet/pods/efb87e6d-3283-469d-9e30-ba78841c8ada/volumes"
Sep 18 20:55:31.509339 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:31.509308    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:55:31.513242 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:55:31.513220    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 20:56:38.712143 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:38.712112    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"]
Sep 18 20:56:38.714225 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:38.712373    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" containerID="cri-o://e66641593e35045248fd3d4b6627a9df09bea8201ce61fb65cc9966bc4f480cd" gracePeriod=30
Sep 18 20:56:38.714225 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:38.712459    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kube-rbac-proxy" containerID="cri-o://172f016cf82b7119e48388c5d116adcd8ccbac55deb27f774b7c33be31e42d24" gracePeriod=30
Sep 18 20:56:39.414504 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:39.414469    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerID="172f016cf82b7119e48388c5d116adcd8ccbac55deb27f774b7c33be31e42d24" exitCode=2
Sep 18 20:56:39.414670 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:39.414507    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" event={"ID":"9c1f8293-a028-472c-828b-64ebbf620ffe","Type":"ContainerDied","Data":"172f016cf82b7119e48388c5d116adcd8ccbac55deb27f774b7c33be31e42d24"}
Sep 18 20:56:40.690955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:40.690916    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 18 20:56:40.695215 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:40.695191    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 18 20:56:43.432190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.432159    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerID="e66641593e35045248fd3d4b6627a9df09bea8201ce61fb65cc9966bc4f480cd" exitCode=0
Sep 18 20:56:43.432529 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.432230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" event={"ID":"9c1f8293-a028-472c-828b-64ebbf620ffe","Type":"ContainerDied","Data":"e66641593e35045248fd3d4b6627a9df09bea8201ce61fb65cc9966bc4f480cd"}
Sep 18 20:56:43.449598 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.449572    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:56:43.623918 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.623841    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c1f8293-a028-472c-828b-64ebbf620ffe-proxy-tls\") pod \"9c1f8293-a028-472c-828b-64ebbf620ffe\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") "
Sep 18 20:56:43.623918 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.623895    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c1f8293-a028-472c-828b-64ebbf620ffe-kserve-provision-location\") pod \"9c1f8293-a028-472c-828b-64ebbf620ffe\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") "
Sep 18 20:56:43.624171 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.623930    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-cabundle-cert\") pod \"9c1f8293-a028-472c-828b-64ebbf620ffe\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") "
Sep 18 20:56:43.624171 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.623947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-rbac-proxy-sar-config\") pod \"9c1f8293-a028-472c-828b-64ebbf620ffe\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") "
Sep 18 20:56:43.624171 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.623979    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7hht\" (UniqueName: \"kubernetes.io/projected/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-api-access-q7hht\") pod \"9c1f8293-a028-472c-828b-64ebbf620ffe\" (UID: \"9c1f8293-a028-472c-828b-64ebbf620ffe\") "
Sep 18 20:56:43.624349 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.624253    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1f8293-a028-472c-828b-64ebbf620ffe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c1f8293-a028-472c-828b-64ebbf620ffe" (UID: "9c1f8293-a028-472c-828b-64ebbf620ffe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:56:43.624407 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.624342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c1f8293-a028-472c-828b-64ebbf620ffe" (UID: "9c1f8293-a028-472c-828b-64ebbf620ffe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:56:43.624407 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.624356    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c1f8293-a028-472c-828b-64ebbf620ffe" (UID: "9c1f8293-a028-472c-828b-64ebbf620ffe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:56:43.625978 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.625957    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1f8293-a028-472c-828b-64ebbf620ffe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c1f8293-a028-472c-828b-64ebbf620ffe" (UID: "9c1f8293-a028-472c-828b-64ebbf620ffe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:56:43.626116 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.626095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-api-access-q7hht" (OuterVolumeSpecName: "kube-api-access-q7hht") pod "9c1f8293-a028-472c-828b-64ebbf620ffe" (UID: "9c1f8293-a028-472c-828b-64ebbf620ffe"). InnerVolumeSpecName "kube-api-access-q7hht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:56:43.725344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.725306    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c1f8293-a028-472c-828b-64ebbf620ffe-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.725344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.725332    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.725344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.725343    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.725344 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.725351    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7hht\" (UniqueName: \"kubernetes.io/projected/9c1f8293-a028-472c-828b-64ebbf620ffe-kube-api-access-q7hht\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.725600 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.725361    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c1f8293-a028-472c-828b-64ebbf620ffe-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:56:43.875588 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875514    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"]
Sep 18 20:56:43.875791 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875780    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container"
Sep 18 20:56:43.875835 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875792    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container"
Sep 18 20:56:43.875835 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875803    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="storage-initializer"
Sep 18 20:56:43.875835 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875809    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="storage-initializer"
Sep 18 20:56:43.875835 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875816    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="storage-initializer"
Sep 18 20:56:43.875835 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875821    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="storage-initializer"
Sep 18 20:56:43.875835 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875832    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kube-rbac-proxy"
Sep 18 20:56:43.875835 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875838    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kube-rbac-proxy"
Sep 18 20:56:43.876074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875847    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kube-rbac-proxy"
Sep 18 20:56:43.876074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875852    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kube-rbac-proxy"
Sep 18 20:56:43.876074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875864    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container"
Sep 18 20:56:43.876074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875868    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container"
Sep 18 20:56:43.876074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875909    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kserve-container"
Sep 18 20:56:43.876074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875917    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kserve-container"
Sep 18 20:56:43.876074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875922    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" containerName="kube-rbac-proxy"
Sep 18 20:56:43.876074 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.875929    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb87e6d-3283-469d-9e30-ba78841c8ada" containerName="kube-rbac-proxy"
Sep 18 20:56:43.878937 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.878915    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:43.881360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.881339    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 18 20:56:43.881465 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.881344    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 20:56:43.888937 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.888917    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"]
Sep 18 20:56:43.924649 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.924619    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"]
Sep 18 20:56:43.927745 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.927730    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:43.930212 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.930193    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 20:56:43.939042 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:43.939021    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"]
Sep 18 20:56:44.028310 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028277    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.028310 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.028503 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e416d2a-afe6-4d30-a896-fe558a1eff01-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.028503 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.028503 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldb5w\" (UniqueName: \"kubernetes.io/projected/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-api-access-ldb5w\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.028503 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e416d2a-afe6-4d30-a896-fe558a1eff01-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.028644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7j65\" (UniqueName: \"kubernetes.io/projected/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-api-access-k7j65\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.028644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.028644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.028644 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.028575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.129479 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.129479 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.129697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e416d2a-afe6-4d30-a896-fe558a1eff01-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.129697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.129697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb5w\" (UniqueName: \"kubernetes.io/projected/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-api-access-ldb5w\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.129697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e416d2a-afe6-4d30-a896-fe558a1eff01-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.129697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7j65\" (UniqueName: \"kubernetes.io/projected/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-api-access-k7j65\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.129697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.129697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.129697 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.130137 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.129951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e416d2a-afe6-4d30-a896-fe558a1eff01-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.130137 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.130126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.130354 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.130330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.130627 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.130354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.130627 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.130518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.130757 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.130680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.132141 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.132119    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e416d2a-afe6-4d30-a896-fe558a1eff01-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.132230 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.132150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.137108 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.137090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7j65\" (UniqueName: \"kubernetes.io/projected/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-api-access-k7j65\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.137211 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.137194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldb5w\" (UniqueName: \"kubernetes.io/projected/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-api-access-ldb5w\") pod \"isvc-canary-stop-predictor-5ffdccb94c-h7bwm\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.189148 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.189123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:44.238139 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.238083    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:44.438878 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.438790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7" event={"ID":"9c1f8293-a028-472c-828b-64ebbf620ffe","Type":"ContainerDied","Data":"532adf5446c707eddc0ab16650508ce88b17aaac58c01d481ee4595552234a59"}
Sep 18 20:56:44.438878 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.438838    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"
Sep 18 20:56:44.438878 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.438873    2566 scope.go:117] "RemoveContainer" containerID="172f016cf82b7119e48388c5d116adcd8ccbac55deb27f774b7c33be31e42d24"
Sep 18 20:56:44.448294 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.448272    2566 scope.go:117] "RemoveContainer" containerID="e66641593e35045248fd3d4b6627a9df09bea8201ce61fb65cc9966bc4f480cd"
Sep 18 20:56:44.457099 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.457080    2566 scope.go:117] "RemoveContainer" containerID="5cc80a7cbfd11e9b46ed2d6d4ae8ab9910d39b59f3f24e5fbd5fc1b45e77bb86"
Sep 18 20:56:44.463332 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.463311    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"]
Sep 18 20:56:44.466930 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.466908    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"]
Sep 18 20:56:44.467662 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:56:44.467639    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode034872a_8d4d_4eff_adea_1d6c3fd86b1c.slice/crio-b1f073c1aa17fd61ee58d5feead30301e76c99ebf59ccec3e711c5318ead47a9 WatchSource:0}: Error finding container b1f073c1aa17fd61ee58d5feead30301e76c99ebf59ccec3e711c5318ead47a9: Status 404 returned error can't find the container with id b1f073c1aa17fd61ee58d5feead30301e76c99ebf59ccec3e711c5318ead47a9
Sep 18 20:56:44.470603 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.470583    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-lptf7"]
Sep 18 20:56:44.512041 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:44.511942    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"]
Sep 18 20:56:44.516007 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:56:44.515984    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e416d2a_afe6_4d30_a896_fe558a1eff01.slice/crio-75c335e74fb411553940a5819dead3886810b7bb7c227aa02ee8bc427bb0ce1a WatchSource:0}: Error finding container 75c335e74fb411553940a5819dead3886810b7bb7c227aa02ee8bc427bb0ce1a: Status 404 returned error can't find the container with id 75c335e74fb411553940a5819dead3886810b7bb7c227aa02ee8bc427bb0ce1a
Sep 18 20:56:45.444336 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:45.444300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" event={"ID":"2e416d2a-afe6-4d30-a896-fe558a1eff01","Type":"ContainerStarted","Data":"9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3"}
Sep 18 20:56:45.444811 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:45.444366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" event={"ID":"2e416d2a-afe6-4d30-a896-fe558a1eff01","Type":"ContainerStarted","Data":"75c335e74fb411553940a5819dead3886810b7bb7c227aa02ee8bc427bb0ce1a"}
Sep 18 20:56:45.445968 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:45.445943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" event={"ID":"e034872a-8d4d-4eff-adea-1d6c3fd86b1c","Type":"ContainerStarted","Data":"f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420"}
Sep 18 20:56:45.446100 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:45.445999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" event={"ID":"e034872a-8d4d-4eff-adea-1d6c3fd86b1c","Type":"ContainerStarted","Data":"b1f073c1aa17fd61ee58d5feead30301e76c99ebf59ccec3e711c5318ead47a9"}
Sep 18 20:56:45.460393 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:45.460372    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c1f8293-a028-472c-828b-64ebbf620ffe" path="/var/lib/kubelet/pods/9c1f8293-a028-472c-828b-64ebbf620ffe/volumes"
Sep 18 20:56:48.461642 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:48.461612    2566 generic.go:356] "Generic (PLEG): container finished" podID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerID="9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3" exitCode=0
Sep 18 20:56:48.461969 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:48.461683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" event={"ID":"2e416d2a-afe6-4d30-a896-fe558a1eff01","Type":"ContainerDied","Data":"9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3"}
Sep 18 20:56:48.463425 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:48.463397    2566 generic.go:356] "Generic (PLEG): container finished" podID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerID="f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420" exitCode=0
Sep 18 20:56:48.463525 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:48.463434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" event={"ID":"e034872a-8d4d-4eff-adea-1d6c3fd86b1c","Type":"ContainerDied","Data":"f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420"}
Sep 18 20:56:49.470299 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:49.470255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" event={"ID":"2e416d2a-afe6-4d30-a896-fe558a1eff01","Type":"ContainerStarted","Data":"61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037"}
Sep 18 20:56:49.470784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:49.470332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" event={"ID":"2e416d2a-afe6-4d30-a896-fe558a1eff01","Type":"ContainerStarted","Data":"e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488"}
Sep 18 20:56:49.470784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:49.470537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:49.472696 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:49.472669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" event={"ID":"e034872a-8d4d-4eff-adea-1d6c3fd86b1c","Type":"ContainerStarted","Data":"388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07"}
Sep 18 20:56:49.472813 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:49.472732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" event={"ID":"e034872a-8d4d-4eff-adea-1d6c3fd86b1c","Type":"ContainerStarted","Data":"76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58"}
Sep 18 20:56:49.472984 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:49.472964    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:49.491238 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:49.491188    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podStartSLOduration=2.810751988 podStartE2EDuration="6.491175476s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.680423488s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:56:49.489078185 +0000 UTC m=+2178.520004327" watchObservedRunningTime="2026-09-18 20:56:49.491175476 +0000 UTC m=+2178.522101616"
Sep 18 20:56:49.510768 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:49.510729    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podStartSLOduration=2.657975551 podStartE2EDuration="6.510716088s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.852740537s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:56:49.509241877 +0000 UTC m=+2178.540168018" watchObservedRunningTime="2026-09-18 20:56:49.510716088 +0000 UTC m=+2178.541642227"
Sep 18 20:56:50.477506 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:50.477472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:50.477876 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:50.477715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:50.478700 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:50.478671    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 18 20:56:50.478818 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:50.478740    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:56:51.480729 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:51.480692    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 18 20:56:51.481101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:51.480892    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:56:56.484741 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:56.484709    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:56:56.485250 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:56.485233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:56:56.485367 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:56.485341    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 18 20:56:56.485701 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:56:56.485682    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:57:06.485397 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:06.485359    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 18 20:57:06.485933 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:06.485591    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:57:16.485765 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:16.485729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:57:16.486248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:16.485728    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 18 20:57:26.485759 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:26.485720    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:57:26.485759 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:26.485753    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 18 20:57:36.485539 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:36.485501    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 18 20:57:36.485921 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:36.485637    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:57:46.485771 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:46.485736    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:57:46.486197 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:46.485736    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 18 20:57:56.486206 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:56.486125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 18 20:57:56.486558 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:57:56.486125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:58:04.455867 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:04.455818    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 18 20:58:06.486609 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:06.486583    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:58:14.457171 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:14.457140    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:58:18.891784 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:18.891747    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"]
Sep 18 20:58:18.892241 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:18.891945    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container" containerID="cri-o://76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58" gracePeriod=30
Sep 18 20:58:18.892241 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:18.892001    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kube-rbac-proxy" containerID="cri-o://388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07" gracePeriod=30
Sep 18 20:58:18.927234 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:18.927208    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"]
Sep 18 20:58:18.927479 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:18.927436    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container" containerID="cri-o://e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488" gracePeriod=30
Sep 18 20:58:18.927560 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:18.927461    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kube-rbac-proxy" containerID="cri-o://61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037" gracePeriod=30
Sep 18 20:58:19.826346 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:19.826313    2566 generic.go:356] "Generic (PLEG): container finished" podID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerID="61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037" exitCode=2
Sep 18 20:58:19.826521 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:19.826385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" event={"ID":"2e416d2a-afe6-4d30-a896-fe558a1eff01","Type":"ContainerDied","Data":"61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037"}
Sep 18 20:58:19.828480 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:19.828459    2566 generic.go:356] "Generic (PLEG): container finished" podID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerID="388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07" exitCode=2
Sep 18 20:58:19.828591 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:19.828529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" event={"ID":"e034872a-8d4d-4eff-adea-1d6c3fd86b1c","Type":"ContainerDied","Data":"388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07"}
Sep 18 20:58:21.480840 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:21.480796    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 18 20:58:21.481245 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:21.480796    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 18 20:58:23.771911 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.771889    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:58:23.838101 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.838077    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:58:23.846329 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.846297    2566 generic.go:356] "Generic (PLEG): container finished" podID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerID="e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488" exitCode=0
Sep 18 20:58:23.846429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.846355    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"
Sep 18 20:58:23.846429 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.846373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" event={"ID":"2e416d2a-afe6-4d30-a896-fe558a1eff01","Type":"ContainerDied","Data":"e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488"}
Sep 18 20:58:23.846546 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.846443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm" event={"ID":"2e416d2a-afe6-4d30-a896-fe558a1eff01","Type":"ContainerDied","Data":"75c335e74fb411553940a5819dead3886810b7bb7c227aa02ee8bc427bb0ce1a"}
Sep 18 20:58:23.846546 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.846488    2566 scope.go:117] "RemoveContainer" containerID="61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037"
Sep 18 20:58:23.848362 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.848342    2566 generic.go:356] "Generic (PLEG): container finished" podID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerID="76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58" exitCode=0
Sep 18 20:58:23.848457 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.848379    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"
Sep 18 20:58:23.848457 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.848403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" event={"ID":"e034872a-8d4d-4eff-adea-1d6c3fd86b1c","Type":"ContainerDied","Data":"76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58"}
Sep 18 20:58:23.848536 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.848455    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd" event={"ID":"e034872a-8d4d-4eff-adea-1d6c3fd86b1c","Type":"ContainerDied","Data":"b1f073c1aa17fd61ee58d5feead30301e76c99ebf59ccec3e711c5318ead47a9"}
Sep 18 20:58:23.854391 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.854374    2566 scope.go:117] "RemoveContainer" containerID="e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488"
Sep 18 20:58:23.862799 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.862770    2566 scope.go:117] "RemoveContainer" containerID="9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3"
Sep 18 20:58:23.870520 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.870502    2566 scope.go:117] "RemoveContainer" containerID="61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037"
Sep 18 20:58:23.870792 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:58:23.870775    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037\": container with ID starting with 61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037 not found: ID does not exist" containerID="61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037"
Sep 18 20:58:23.870879 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.870799    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037"} err="failed to get container status \"61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037\": rpc error: code = NotFound desc = could not find container \"61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037\": container with ID starting with 61101ee3a12b5ebedd31fc2497ff2d6164697e4eeee8caf8ac5ec380af6bd037 not found: ID does not exist"
Sep 18 20:58:23.870879 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.870815    2566 scope.go:117] "RemoveContainer" containerID="e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488"
Sep 18 20:58:23.871080 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:58:23.871062    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488\": container with ID starting with e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488 not found: ID does not exist" containerID="e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488"
Sep 18 20:58:23.871169 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.871084    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488"} err="failed to get container status \"e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488\": rpc error: code = NotFound desc = could not find container \"e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488\": container with ID starting with e0da9808e36769a3f199092538ca20600c65852d059a3a020239a9c5d14ec488 not found: ID does not exist"
Sep 18 20:58:23.871169 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.871097    2566 scope.go:117] "RemoveContainer" containerID="9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3"
Sep 18 20:58:23.871307 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:58:23.871291    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3\": container with ID starting with 9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3 not found: ID does not exist" containerID="9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3"
Sep 18 20:58:23.871370 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.871308    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3"} err="failed to get container status \"9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3\": rpc error: code = NotFound desc = could not find container \"9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3\": container with ID starting with 9907afd42615c3676c4d21fd023e8426930ea174a1ac5c9b9aec3dc07861d2f3 not found: ID does not exist"
Sep 18 20:58:23.871370 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.871321    2566 scope.go:117] "RemoveContainer" containerID="388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07"
Sep 18 20:58:23.878968 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.878943    2566 scope.go:117] "RemoveContainer" containerID="76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58"
Sep 18 20:58:23.885857 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.885842    2566 scope.go:117] "RemoveContainer" containerID="f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420"
Sep 18 20:58:23.892668 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.892653    2566 scope.go:117] "RemoveContainer" containerID="388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07"
Sep 18 20:58:23.892920 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:58:23.892905    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07\": container with ID starting with 388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07 not found: ID does not exist" containerID="388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07"
Sep 18 20:58:23.892978 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.892924    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07"} err="failed to get container status \"388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07\": rpc error: code = NotFound desc = could not find container \"388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07\": container with ID starting with 388a29d9f0d071f59c7f9bd194f35a92c5cf552fa4e355225eaaea479d7a0a07 not found: ID does not exist"
Sep 18 20:58:23.892978 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.892938    2566 scope.go:117] "RemoveContainer" containerID="76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58"
Sep 18 20:58:23.893175 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:58:23.893159    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58\": container with ID starting with 76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58 not found: ID does not exist" containerID="76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58"
Sep 18 20:58:23.893227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.893188    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58"} err="failed to get container status \"76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58\": rpc error: code = NotFound desc = could not find container \"76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58\": container with ID starting with 76af4b51287c94dda4ce93cede2c8f6f493bb99f536afbdc95e5524e32a49d58 not found: ID does not exist"
Sep 18 20:58:23.893227 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.893202    2566 scope.go:117] "RemoveContainer" containerID="f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420"
Sep 18 20:58:23.893424 ip-10-0-143-76 kubenswrapper[2566]: E0918 20:58:23.893404    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420\": container with ID starting with f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420 not found: ID does not exist" containerID="f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420"
Sep 18 20:58:23.893481 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.893437    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420"} err="failed to get container status \"f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420\": rpc error: code = NotFound desc = could not find container \"f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420\": container with ID starting with f6898effdf74c1db7ab8dbfc422589d2cf7bc9e2300dce8be4764e7a25090420 not found: ID does not exist"
Sep 18 20:58:23.918305 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918283    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-rbac-proxy-sar-config\") pod \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") "
Sep 18 20:58:23.918406 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918327    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-cabundle-cert\") pod \"2e416d2a-afe6-4d30-a896-fe558a1eff01\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") "
Sep 18 20:58:23.918406 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918346    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e416d2a-afe6-4d30-a896-fe558a1eff01-kserve-provision-location\") pod \"2e416d2a-afe6-4d30-a896-fe558a1eff01\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") "
Sep 18 20:58:23.918406 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918371    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldb5w\" (UniqueName: \"kubernetes.io/projected/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-api-access-ldb5w\") pod \"2e416d2a-afe6-4d30-a896-fe558a1eff01\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") "
Sep 18 20:58:23.918406 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7j65\" (UniqueName: \"kubernetes.io/projected/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-api-access-k7j65\") pod \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") "
Sep 18 20:58:23.918634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-proxy-tls\") pod \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") "
Sep 18 20:58:23.918634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918455    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-rbac-proxy-sar-config\") pod \"2e416d2a-afe6-4d30-a896-fe558a1eff01\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") "
Sep 18 20:58:23.918634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kserve-provision-location\") pod \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") "
Sep 18 20:58:23.918634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918523    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e416d2a-afe6-4d30-a896-fe558a1eff01-proxy-tls\") pod \"2e416d2a-afe6-4d30-a896-fe558a1eff01\" (UID: \"2e416d2a-afe6-4d30-a896-fe558a1eff01\") "
Sep 18 20:58:23.918634 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918570    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-cabundle-cert\") pod \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\" (UID: \"e034872a-8d4d-4eff-adea-1d6c3fd86b1c\") "
Sep 18 20:58:23.918870 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918689    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e034872a-8d4d-4eff-adea-1d6c3fd86b1c" (UID: "e034872a-8d4d-4eff-adea-1d6c3fd86b1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:58:23.918870 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918715    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e416d2a-afe6-4d30-a896-fe558a1eff01" (UID: "2e416d2a-afe6-4d30-a896-fe558a1eff01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:58:23.918870 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e416d2a-afe6-4d30-a896-fe558a1eff01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e416d2a-afe6-4d30-a896-fe558a1eff01" (UID: "2e416d2a-afe6-4d30-a896-fe558a1eff01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:58:23.918993 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.918870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e034872a-8d4d-4eff-adea-1d6c3fd86b1c" (UID: "e034872a-8d4d-4eff-adea-1d6c3fd86b1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:58:23.919078 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.919046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e034872a-8d4d-4eff-adea-1d6c3fd86b1c" (UID: "e034872a-8d4d-4eff-adea-1d6c3fd86b1c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:58:23.919078 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.919067    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e416d2a-afe6-4d30-a896-fe558a1eff01-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:23.919190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.919089    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:23.919190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.919107    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:23.919190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.919116    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e416d2a-afe6-4d30-a896-fe558a1eff01" (UID: "2e416d2a-afe6-4d30-a896-fe558a1eff01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:58:23.919190 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.919124    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:23.920788 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.920761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-api-access-k7j65" (OuterVolumeSpecName: "kube-api-access-k7j65") pod "e034872a-8d4d-4eff-adea-1d6c3fd86b1c" (UID: "e034872a-8d4d-4eff-adea-1d6c3fd86b1c"). InnerVolumeSpecName "kube-api-access-k7j65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:58:23.920882 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.920812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e416d2a-afe6-4d30-a896-fe558a1eff01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e416d2a-afe6-4d30-a896-fe558a1eff01" (UID: "2e416d2a-afe6-4d30-a896-fe558a1eff01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:58:23.920882 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.920812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e034872a-8d4d-4eff-adea-1d6c3fd86b1c" (UID: "e034872a-8d4d-4eff-adea-1d6c3fd86b1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:58:23.920956 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:23.920887    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-api-access-ldb5w" (OuterVolumeSpecName: "kube-api-access-ldb5w") pod "2e416d2a-afe6-4d30-a896-fe558a1eff01" (UID: "2e416d2a-afe6-4d30-a896-fe558a1eff01"). InnerVolumeSpecName "kube-api-access-ldb5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:58:24.020341 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.020297    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldb5w\" (UniqueName: \"kubernetes.io/projected/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-api-access-ldb5w\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:24.020341 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.020336    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7j65\" (UniqueName: \"kubernetes.io/projected/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-kube-api-access-k7j65\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:24.020341 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.020345    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:24.020341 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.020355    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e416d2a-afe6-4d30-a896-fe558a1eff01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:24.020596 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.020363    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e416d2a-afe6-4d30-a896-fe558a1eff01-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:24.020596 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.020372    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e034872a-8d4d-4eff-adea-1d6c3fd86b1c-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 20:58:24.172714 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.172684    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"]
Sep 18 20:58:24.176408 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.176387    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-h7bwm"]
Sep 18 20:58:24.186900 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.186877    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"]
Sep 18 20:58:24.188360 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:24.188339    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-9c6gd"]
Sep 18 20:58:25.460109 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:25.460072    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" path="/var/lib/kubelet/pods/2e416d2a-afe6-4d30-a896-fe558a1eff01/volumes"
Sep 18 20:58:25.460800 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:25.460773    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" path="/var/lib/kubelet/pods/e034872a-8d4d-4eff-adea-1d6c3fd86b1c/volumes"
Sep 18 20:58:29.165555 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165524    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"]
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165807    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165822    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165834    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="storage-initializer"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165840    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="storage-initializer"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165845    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kube-rbac-proxy"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165851    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kube-rbac-proxy"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165857    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="storage-initializer"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165862    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="storage-initializer"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165874    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kube-rbac-proxy"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165879    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kube-rbac-proxy"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165886    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165892    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165932    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kube-rbac-proxy"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165941    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kserve-container"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165946    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e416d2a-afe6-4d30-a896-fe558a1eff01" containerName="kube-rbac-proxy"
Sep 18 20:58:29.165951 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.165953    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e034872a-8d4d-4eff-adea-1d6c3fd86b1c" containerName="kserve-container"
Sep 18 20:58:29.170732 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.170714    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.173492 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.173469    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 20:58:29.173851 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.173831    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:58:29.173944 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.173866    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:58:29.173944 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.173872    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:58:29.173944 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.173887    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 18 20:58:29.173944 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.173900    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:58:29.174290 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.174277    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 18 20:58:29.181492 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.181471    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"]
Sep 18 20:58:29.261387 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.261360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/995b6994-306e-4d79-87a5-5ff896e943df-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.261529 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.261395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/995b6994-306e-4d79-87a5-5ff896e943df-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.261529 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.261418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2lpn\" (UniqueName: \"kubernetes.io/projected/995b6994-306e-4d79-87a5-5ff896e943df-kube-api-access-g2lpn\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.261529 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.261439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.261657 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.261540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.362340 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.362310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.362474 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.362376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/995b6994-306e-4d79-87a5-5ff896e943df-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.362474 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.362409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/995b6994-306e-4d79-87a5-5ff896e943df-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.362474 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.362439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2lpn\" (UniqueName: \"kubernetes.io/projected/995b6994-306e-4d79-87a5-5ff896e943df-kube-api-access-g2lpn\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.362592 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.362559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.362789 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.362760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/995b6994-306e-4d79-87a5-5ff896e943df-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.362858 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.362836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.363167 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.363151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.364805 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.364789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/995b6994-306e-4d79-87a5-5ff896e943df-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.369943 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.369922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2lpn\" (UniqueName: \"kubernetes.io/projected/995b6994-306e-4d79-87a5-5ff896e943df-kube-api-access-g2lpn\") pod \"isvc-lightgbm-predictor-54f85ddf88-h6d7r\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.481662 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.481578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:29.751248 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.751223    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"]
Sep 18 20:58:29.753324 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:58:29.753298    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995b6994_306e_4d79_87a5_5ff896e943df.slice/crio-30385549fe849ab1de420e727aa1e2f5bf9611cef12dda5168a691ca90d013b3 WatchSource:0}: Error finding container 30385549fe849ab1de420e727aa1e2f5bf9611cef12dda5168a691ca90d013b3: Status 404 returned error can't find the container with id 30385549fe849ab1de420e727aa1e2f5bf9611cef12dda5168a691ca90d013b3
Sep 18 20:58:29.875184 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.875149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" event={"ID":"995b6994-306e-4d79-87a5-5ff896e943df","Type":"ContainerStarted","Data":"386a73557b5ac8e2a9b0e12a9cc75bff166723b1e88607d3c73a4ce24d23271a"}
Sep 18 20:58:29.875359 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:29.875224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" event={"ID":"995b6994-306e-4d79-87a5-5ff896e943df","Type":"ContainerStarted","Data":"30385549fe849ab1de420e727aa1e2f5bf9611cef12dda5168a691ca90d013b3"}
Sep 18 20:58:33.895099 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:33.895071    2566 generic.go:356] "Generic (PLEG): container finished" podID="995b6994-306e-4d79-87a5-5ff896e943df" containerID="386a73557b5ac8e2a9b0e12a9cc75bff166723b1e88607d3c73a4ce24d23271a" exitCode=0
Sep 18 20:58:33.895415 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:33.895107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" event={"ID":"995b6994-306e-4d79-87a5-5ff896e943df","Type":"ContainerDied","Data":"386a73557b5ac8e2a9b0e12a9cc75bff166723b1e88607d3c73a4ce24d23271a"}
Sep 18 20:58:33.896765 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:33.896746    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:58:40.928132 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:40.928095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" event={"ID":"995b6994-306e-4d79-87a5-5ff896e943df","Type":"ContainerStarted","Data":"4bf07ba72067bcd7d98b9fef7ed7d14691bd441e48b4e85070273238280f034a"}
Sep 18 20:58:40.928598 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:40.928164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" event={"ID":"995b6994-306e-4d79-87a5-5ff896e943df","Type":"ContainerStarted","Data":"f42d8b1983b5a661378ff563cef1b75bdc1a7e9cc34f42787fcae01a840b4d2c"}
Sep 18 20:58:40.928598 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:40.928373    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:40.948641 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:40.948601    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podStartSLOduration=1.588761137 podStartE2EDuration="11.94858835s" totalImagesPullingTime="6.325741741s" totalInitContainerRuntime="4.034085472s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:58:40.946297462 +0000 UTC m=+2289.977223616" watchObservedRunningTime="2026-09-18 20:58:40.94858835 +0000 UTC m=+2289.979514489"
Sep 18 20:58:41.932795 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:41.932759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:41.934100 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:41.934074    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 20:58:42.936329 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:42.936288    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 20:58:47.940801 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:47.940774    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 20:58:47.941228 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:47.941203    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 20:58:57.942176 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:58:57.942136    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 20:59:02.858435 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.858395    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"]
Sep 18 20:59:02.861955 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.861938    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:02.864518 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.864492    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 18 20:59:02.864598 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.864493    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:59:02.872651 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.872622    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"]
Sep 18 20:59:02.915070 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.915046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:02.915191 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.915074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:02.915191 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.915093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6341399-06fa-4733-97d2-aa6c141e81c4-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:02.915191 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.915109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwsbd\" (UniqueName: \"kubernetes.io/projected/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-api-access-zwsbd\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:02.915311 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:02.915203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6341399-06fa-4733-97d2-aa6c141e81c4-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.016455 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.016426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6341399-06fa-4733-97d2-aa6c141e81c4-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.016614 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.016495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.016614 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.016516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.016614 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.016541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6341399-06fa-4733-97d2-aa6c141e81c4-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.016614 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.016556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwsbd\" (UniqueName: \"kubernetes.io/projected/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-api-access-zwsbd\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.016926 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.016888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6341399-06fa-4733-97d2-aa6c141e81c4-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.017224 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.017205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.017313 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.017285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.018924 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.018906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6341399-06fa-4733-97d2-aa6c141e81c4-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.024488 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.024470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwsbd\" (UniqueName: \"kubernetes.io/projected/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-api-access-zwsbd\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-sgbjm\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.173266 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.173193    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:03.443825 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:03.443716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"]
Sep 18 20:59:03.446410 ip-10-0-143-76 kubenswrapper[2566]: W0918 20:59:03.446387    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6341399_06fa_4733_97d2_aa6c141e81c4.slice/crio-f6d7b3300709489a77822277cc43ed19631c2c5b6612a9de2169d1e0403a8a2c WatchSource:0}: Error finding container f6d7b3300709489a77822277cc43ed19631c2c5b6612a9de2169d1e0403a8a2c: Status 404 returned error can't find the container with id f6d7b3300709489a77822277cc43ed19631c2c5b6612a9de2169d1e0403a8a2c
Sep 18 20:59:04.012184 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:04.012146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" event={"ID":"d6341399-06fa-4733-97d2-aa6c141e81c4","Type":"ContainerStarted","Data":"1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd"}
Sep 18 20:59:04.012643 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:04.012204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" event={"ID":"d6341399-06fa-4733-97d2-aa6c141e81c4","Type":"ContainerStarted","Data":"f6d7b3300709489a77822277cc43ed19631c2c5b6612a9de2169d1e0403a8a2c"}
Sep 18 20:59:07.941229 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:07.941189    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 20:59:08.028859 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:08.028829    2566 generic.go:356] "Generic (PLEG): container finished" podID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerID="1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd" exitCode=0
Sep 18 20:59:08.029083 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:08.028879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" event={"ID":"d6341399-06fa-4733-97d2-aa6c141e81c4","Type":"ContainerDied","Data":"1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd"}
Sep 18 20:59:17.942053 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:17.941982    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 20:59:27.941499 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:27.941461    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 20:59:28.119577 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:28.119545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" event={"ID":"d6341399-06fa-4733-97d2-aa6c141e81c4","Type":"ContainerStarted","Data":"6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609"}
Sep 18 20:59:28.119767 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:28.119606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" event={"ID":"d6341399-06fa-4733-97d2-aa6c141e81c4","Type":"ContainerStarted","Data":"56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4"}
Sep 18 20:59:28.119822 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:28.119799    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:28.138104 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:28.138061    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podStartSLOduration=3.275857234 podStartE2EDuration="26.138047922s" totalImagesPullingTime="19.055694696s" totalInitContainerRuntime="3.806495992s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:59:28.136563992 +0000 UTC m=+2337.167490132" watchObservedRunningTime="2026-09-18 20:59:28.138047922 +0000 UTC m=+2337.168974061"
Sep 18 20:59:29.123801 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:29.123768    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:29.124811 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:29.124782    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 18 20:59:30.128191 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:30.128144    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 18 20:59:35.132503 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:35.132477    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 20:59:35.132943 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:35.132916    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 18 20:59:37.941831 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:37.941789    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 20:59:45.133790 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:45.133749    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 18 20:59:47.941267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:47.941230    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 20:59:55.133061 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:55.133001    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 18 20:59:57.941267 ip-10-0-143-76 kubenswrapper[2566]: I0918 20:59:57.941229    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 21:00:01.464934 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:01.464896    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 21:00:05.133278 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:05.133240    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 18 21:00:11.465363 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:11.465325    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 21:00:15.133098 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:15.133062    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 18 21:00:21.466134 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:21.466106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 21:00:25.133232 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:25.133193    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 18 21:00:29.976149 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:29.976104    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"]
Sep 18 21:00:29.976738 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:29.976371    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" containerID="cri-o://f42d8b1983b5a661378ff563cef1b75bdc1a7e9cc34f42787fcae01a840b4d2c" gracePeriod=30
Sep 18 21:00:29.976738 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:29.976420    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kube-rbac-proxy" containerID="cri-o://4bf07ba72067bcd7d98b9fef7ed7d14691bd441e48b4e85070273238280f034a" gracePeriod=30
Sep 18 21:00:30.366673 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:30.366634    2566 generic.go:356] "Generic (PLEG): container finished" podID="995b6994-306e-4d79-87a5-5ff896e943df" containerID="4bf07ba72067bcd7d98b9fef7ed7d14691bd441e48b4e85070273238280f034a" exitCode=2
Sep 18 21:00:30.366872 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:30.366701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" event={"ID":"995b6994-306e-4d79-87a5-5ff896e943df","Type":"ContainerDied","Data":"4bf07ba72067bcd7d98b9fef7ed7d14691bd441e48b4e85070273238280f034a"}
Sep 18 21:00:31.465606 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:31.465572    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 18 21:00:31.534870 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:31.534841    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 21:00:31.539514 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:31.539492    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 21:00:32.937224 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:32.937181    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Sep 18 21:00:35.133279 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.133187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 21:00:35.217442 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.217411    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"]
Sep 18 21:00:35.219709 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.219692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.230731 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.230715    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 18 21:00:35.230851 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.230714    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 18 21:00:35.243027 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.242996    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"]
Sep 18 21:00:35.377134 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.377108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.377306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.377164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.377306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.377211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx9qb\" (UniqueName: \"kubernetes.io/projected/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-api-access-rx9qb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.377306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.377236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ce29a2-a52d-4eef-852b-cf95d02c3432-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.377306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.377261    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.395391 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.395320    2566 generic.go:356] "Generic (PLEG): container finished" podID="995b6994-306e-4d79-87a5-5ff896e943df" containerID="f42d8b1983b5a661378ff563cef1b75bdc1a7e9cc34f42787fcae01a840b4d2c" exitCode=0
Sep 18 21:00:35.395391 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.395372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" event={"ID":"995b6994-306e-4d79-87a5-5ff896e943df","Type":"ContainerDied","Data":"f42d8b1983b5a661378ff563cef1b75bdc1a7e9cc34f42787fcae01a840b4d2c"}
Sep 18 21:00:35.412059 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.412037    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 21:00:35.477655 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.477622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.477811 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.477672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx9qb\" (UniqueName: \"kubernetes.io/projected/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-api-access-rx9qb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.477811 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.477697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ce29a2-a52d-4eef-852b-cf95d02c3432-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.477811 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.477720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.477811 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.477794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.478058 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.477957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.478453 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.478430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.478539 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.478458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.480025 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.479998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ce29a2-a52d-4eef-852b-cf95d02c3432-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.486426 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.486398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx9qb\" (UniqueName: \"kubernetes.io/projected/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-api-access-rx9qb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.529658 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.529639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:35.578461 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.578422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/995b6994-306e-4d79-87a5-5ff896e943df-proxy-tls\") pod \"995b6994-306e-4d79-87a5-5ff896e943df\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") "
Sep 18 21:00:35.578589 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.578478    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-kube-rbac-proxy-sar-config\") pod \"995b6994-306e-4d79-87a5-5ff896e943df\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") "
Sep 18 21:00:35.578589 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.578508    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-cabundle-cert\") pod \"995b6994-306e-4d79-87a5-5ff896e943df\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") "
Sep 18 21:00:35.578589 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.578575    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2lpn\" (UniqueName: \"kubernetes.io/projected/995b6994-306e-4d79-87a5-5ff896e943df-kube-api-access-g2lpn\") pod \"995b6994-306e-4d79-87a5-5ff896e943df\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") "
Sep 18 21:00:35.578865 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.578615    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/995b6994-306e-4d79-87a5-5ff896e943df-kserve-provision-location\") pod \"995b6994-306e-4d79-87a5-5ff896e943df\" (UID: \"995b6994-306e-4d79-87a5-5ff896e943df\") "
Sep 18 21:00:35.579354 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.579209    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995b6994-306e-4d79-87a5-5ff896e943df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "995b6994-306e-4d79-87a5-5ff896e943df" (UID: "995b6994-306e-4d79-87a5-5ff896e943df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:00:35.579354 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.579297    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "995b6994-306e-4d79-87a5-5ff896e943df" (UID: "995b6994-306e-4d79-87a5-5ff896e943df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:00:35.579354 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.579308    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "995b6994-306e-4d79-87a5-5ff896e943df" (UID: "995b6994-306e-4d79-87a5-5ff896e943df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:00:35.581378 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.581343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995b6994-306e-4d79-87a5-5ff896e943df-kube-api-access-g2lpn" (OuterVolumeSpecName: "kube-api-access-g2lpn") pod "995b6994-306e-4d79-87a5-5ff896e943df" (UID: "995b6994-306e-4d79-87a5-5ff896e943df"). InnerVolumeSpecName "kube-api-access-g2lpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:00:35.581737 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.581715    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995b6994-306e-4d79-87a5-5ff896e943df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "995b6994-306e-4d79-87a5-5ff896e943df" (UID: "995b6994-306e-4d79-87a5-5ff896e943df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:00:35.680302 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.680213    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/995b6994-306e-4d79-87a5-5ff896e943df-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:35.680302 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.680255    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:35.680302 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.680265    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/995b6994-306e-4d79-87a5-5ff896e943df-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:35.680302 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.680275    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2lpn\" (UniqueName: \"kubernetes.io/projected/995b6994-306e-4d79-87a5-5ff896e943df-kube-api-access-g2lpn\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:35.680302 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.680283    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/995b6994-306e-4d79-87a5-5ff896e943df-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:35.801123 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:35.801081    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"]
Sep 18 21:00:35.803230 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:00:35.803205    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ce29a2_a52d_4eef_852b_cf95d02c3432.slice/crio-fb88495e81cec17c537cacba608ced3a6c3ff1561fefec8c06c8b19af2840fab WatchSource:0}: Error finding container fb88495e81cec17c537cacba608ced3a6c3ff1561fefec8c06c8b19af2840fab: Status 404 returned error can't find the container with id fb88495e81cec17c537cacba608ced3a6c3ff1561fefec8c06c8b19af2840fab
Sep 18 21:00:36.400676 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:36.400638    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" event={"ID":"f4ce29a2-a52d-4eef-852b-cf95d02c3432","Type":"ContainerStarted","Data":"9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b"}
Sep 18 21:00:36.401143 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:36.400707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" event={"ID":"f4ce29a2-a52d-4eef-852b-cf95d02c3432","Type":"ContainerStarted","Data":"fb88495e81cec17c537cacba608ced3a6c3ff1561fefec8c06c8b19af2840fab"}
Sep 18 21:00:36.402545 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:36.402522    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r" event={"ID":"995b6994-306e-4d79-87a5-5ff896e943df","Type":"ContainerDied","Data":"30385549fe849ab1de420e727aa1e2f5bf9611cef12dda5168a691ca90d013b3"}
Sep 18 21:00:36.402678 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:36.402566    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"
Sep 18 21:00:36.402678 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:36.402582    2566 scope.go:117] "RemoveContainer" containerID="4bf07ba72067bcd7d98b9fef7ed7d14691bd441e48b4e85070273238280f034a"
Sep 18 21:00:36.411030 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:36.410999    2566 scope.go:117] "RemoveContainer" containerID="f42d8b1983b5a661378ff563cef1b75bdc1a7e9cc34f42787fcae01a840b4d2c"
Sep 18 21:00:36.420204 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:36.420185    2566 scope.go:117] "RemoveContainer" containerID="386a73557b5ac8e2a9b0e12a9cc75bff166723b1e88607d3c73a4ce24d23271a"
Sep 18 21:00:36.431351 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:36.431330    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"]
Sep 18 21:00:36.435200 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:36.435180    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h6d7r"]
Sep 18 21:00:37.459872 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:37.459841    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="995b6994-306e-4d79-87a5-5ff896e943df" path="/var/lib/kubelet/pods/995b6994-306e-4d79-87a5-5ff896e943df/volumes"
Sep 18 21:00:40.420740 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:40.420705    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerID="9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b" exitCode=0
Sep 18 21:00:40.421184 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:40.420758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" event={"ID":"f4ce29a2-a52d-4eef-852b-cf95d02c3432","Type":"ContainerDied","Data":"9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b"}
Sep 18 21:00:41.430630 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:41.430600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" event={"ID":"f4ce29a2-a52d-4eef-852b-cf95d02c3432","Type":"ContainerStarted","Data":"cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5"}
Sep 18 21:00:41.431031 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:41.430660    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" event={"ID":"f4ce29a2-a52d-4eef-852b-cf95d02c3432","Type":"ContainerStarted","Data":"dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0"}
Sep 18 21:00:41.431031 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:41.430859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:41.450704 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:41.450660    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podStartSLOduration=2.527872894 podStartE2EDuration="6.450644585s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.922771691s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:00:41.449666554 +0000 UTC m=+2410.480592694" watchObservedRunningTime="2026-09-18 21:00:41.450644585 +0000 UTC m=+2410.481570724"
Sep 18 21:00:42.434414 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:42.434386    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:42.435857 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:42.435829    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:00:43.439148 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:43.439110    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:00:43.587090 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:43.587061    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"]
Sep 18 21:00:43.587347 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:43.587322    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" containerID="cri-o://56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4" gracePeriod=30
Sep 18 21:00:43.587476 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:43.587348    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kube-rbac-proxy" containerID="cri-o://6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609" gracePeriod=30
Sep 18 21:00:44.445276 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:44.445245    2566 generic.go:356] "Generic (PLEG): container finished" podID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerID="6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609" exitCode=2
Sep 18 21:00:44.445641 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:44.445312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" event={"ID":"d6341399-06fa-4733-97d2-aa6c141e81c4","Type":"ContainerDied","Data":"6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609"}
Sep 18 21:00:45.129220 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:45.129174    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 18 21:00:45.133710 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:45.133684    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 18 21:00:47.731184 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.731160    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 21:00:47.789271 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.789241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-rbac-proxy-sar-config\") pod \"d6341399-06fa-4733-97d2-aa6c141e81c4\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") "
Sep 18 21:00:47.789408 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.789280    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6341399-06fa-4733-97d2-aa6c141e81c4-kserve-provision-location\") pod \"d6341399-06fa-4733-97d2-aa6c141e81c4\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") "
Sep 18 21:00:47.789408 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.789336    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6341399-06fa-4733-97d2-aa6c141e81c4-proxy-tls\") pod \"d6341399-06fa-4733-97d2-aa6c141e81c4\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") "
Sep 18 21:00:47.789408 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.789362    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-cabundle-cert\") pod \"d6341399-06fa-4733-97d2-aa6c141e81c4\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") "
Sep 18 21:00:47.789408 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.789410    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwsbd\" (UniqueName: \"kubernetes.io/projected/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-api-access-zwsbd\") pod \"d6341399-06fa-4733-97d2-aa6c141e81c4\" (UID: \"d6341399-06fa-4733-97d2-aa6c141e81c4\") "
Sep 18 21:00:47.789648 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.789625    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6341399-06fa-4733-97d2-aa6c141e81c4" (UID: "d6341399-06fa-4733-97d2-aa6c141e81c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:00:47.789707 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.789678    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6341399-06fa-4733-97d2-aa6c141e81c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6341399-06fa-4733-97d2-aa6c141e81c4" (UID: "d6341399-06fa-4733-97d2-aa6c141e81c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:00:47.789767 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.789745    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6341399-06fa-4733-97d2-aa6c141e81c4" (UID: "d6341399-06fa-4733-97d2-aa6c141e81c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:00:47.791353 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.791336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6341399-06fa-4733-97d2-aa6c141e81c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6341399-06fa-4733-97d2-aa6c141e81c4" (UID: "d6341399-06fa-4733-97d2-aa6c141e81c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:00:47.791496 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.791476    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-api-access-zwsbd" (OuterVolumeSpecName: "kube-api-access-zwsbd") pod "d6341399-06fa-4733-97d2-aa6c141e81c4" (UID: "d6341399-06fa-4733-97d2-aa6c141e81c4"). InnerVolumeSpecName "kube-api-access-zwsbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:00:47.890525 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.890462    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:47.890525 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.890483    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwsbd\" (UniqueName: \"kubernetes.io/projected/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-api-access-zwsbd\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:47.890525 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.890493    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6341399-06fa-4733-97d2-aa6c141e81c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:47.890525 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.890503    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6341399-06fa-4733-97d2-aa6c141e81c4-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:47.890525 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:47.890512    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6341399-06fa-4733-97d2-aa6c141e81c4-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:00:48.444164 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.444135    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:00:48.444659 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.444626    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:00:48.465968 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.465941    2566 generic.go:356] "Generic (PLEG): container finished" podID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerID="56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4" exitCode=0
Sep 18 21:00:48.466133 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.465993    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"
Sep 18 21:00:48.466133 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.466006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" event={"ID":"d6341399-06fa-4733-97d2-aa6c141e81c4","Type":"ContainerDied","Data":"56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4"}
Sep 18 21:00:48.466133 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.466077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm" event={"ID":"d6341399-06fa-4733-97d2-aa6c141e81c4","Type":"ContainerDied","Data":"f6d7b3300709489a77822277cc43ed19631c2c5b6612a9de2169d1e0403a8a2c"}
Sep 18 21:00:48.466133 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.466117    2566 scope.go:117] "RemoveContainer" containerID="6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609"
Sep 18 21:00:48.475151 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.475133    2566 scope.go:117] "RemoveContainer" containerID="56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4"
Sep 18 21:00:48.482358 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.482343    2566 scope.go:117] "RemoveContainer" containerID="1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd"
Sep 18 21:00:48.487977 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.487952    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"]
Sep 18 21:00:48.489997 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.489982    2566 scope.go:117] "RemoveContainer" containerID="6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609"
Sep 18 21:00:48.490300 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:00:48.490280    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609\": container with ID starting with 6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609 not found: ID does not exist" containerID="6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609"
Sep 18 21:00:48.490396 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.490311    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609"} err="failed to get container status \"6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609\": rpc error: code = NotFound desc = could not find container \"6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609\": container with ID starting with 6122fd9278a6728bf2045ba5cd984046880b8caca9e65e109029c7b6f4267609 not found: ID does not exist"
Sep 18 21:00:48.490396 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.490333    2566 scope.go:117] "RemoveContainer" containerID="56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4"
Sep 18 21:00:48.490609 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:00:48.490592    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4\": container with ID starting with 56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4 not found: ID does not exist" containerID="56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4"
Sep 18 21:00:48.490652 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.490615    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4"} err="failed to get container status \"56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4\": rpc error: code = NotFound desc = could not find container \"56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4\": container with ID starting with 56eb7c5ad18c1c348d7e6a8b3ea9a77d40196ce13309f3c06cfe9c4119f12db4 not found: ID does not exist"
Sep 18 21:00:48.490652 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.490632    2566 scope.go:117] "RemoveContainer" containerID="1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd"
Sep 18 21:00:48.490893 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:00:48.490877    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd\": container with ID starting with 1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd not found: ID does not exist" containerID="1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd"
Sep 18 21:00:48.490932 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.490898    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd"} err="failed to get container status \"1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd\": rpc error: code = NotFound desc = could not find container \"1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd\": container with ID starting with 1e11141ca84a05c3991f746dc4c5de853b59391c3bb537b0dd4034438755bfdd not found: ID does not exist"
Sep 18 21:00:48.491147 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:48.491130    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-sgbjm"]
Sep 18 21:00:49.460693 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:49.460663    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" path="/var/lib/kubelet/pods/d6341399-06fa-4733-97d2-aa6c141e81c4/volumes"
Sep 18 21:00:58.444855 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:00:58.444764    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:01:08.445284 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:08.445240    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:01:18.444692 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:18.444650    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:01:28.445192 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:28.445152    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:01:38.444542 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:38.444502    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:01:48.444806 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:48.444763    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:01:54.048997 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.048961    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"]
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049282    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="storage-initializer"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049295    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="storage-initializer"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049304    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049309    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049322    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="storage-initializer"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049328    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="storage-initializer"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049338    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049343    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049351    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kube-rbac-proxy"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049357    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kube-rbac-proxy"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049367    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kube-rbac-proxy"
Sep 18 21:01:54.049374 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049371    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kube-rbac-proxy"
Sep 18 21:01:54.049745 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049417    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kserve-container"
Sep 18 21:01:54.049745 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049424    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6341399-06fa-4733-97d2-aa6c141e81c4" containerName="kube-rbac-proxy"
Sep 18 21:01:54.049745 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049431    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kserve-container"
Sep 18 21:01:54.049745 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.049436    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="995b6994-306e-4d79-87a5-5ff896e943df" containerName="kube-rbac-proxy"
Sep 18 21:01:54.052648 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.052632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.055388 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.055368    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 18 21:01:54.055506 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.055433    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 18 21:01:54.064233 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.064210    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"]
Sep 18 21:01:54.208200 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.208167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.208379 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.208207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.208379 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.208303    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmj95\" (UniqueName: \"kubernetes.io/projected/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-api-access-zmj95\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.208379 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.208332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.208379 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.208360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015b3fcf-8ede-44c2-9290-2ee095f1af0c-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.308830 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.308751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.308830 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.308794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmj95\" (UniqueName: \"kubernetes.io/projected/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-api-access-zmj95\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.308830 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.308811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.309106 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.308840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015b3fcf-8ede-44c2-9290-2ee095f1af0c-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.309106 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.308912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.309303 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.309281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.309543 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.309520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.309543 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.309536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.311408 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.311393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015b3fcf-8ede-44c2-9290-2ee095f1af0c-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.317488 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.317458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmj95\" (UniqueName: \"kubernetes.io/projected/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-api-access-zmj95\") pod \"isvc-xgboost-v2-predictor-fcc668bc-5db4w\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.362435 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.362412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:54.633296 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.633267    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"]
Sep 18 21:01:54.635485 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:01:54.635457    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015b3fcf_8ede_44c2_9290_2ee095f1af0c.slice/crio-c797ddbd7447e7056e6ec2a34c981ab16b3b09f9f47e4b1cf6b09c5cf95b4e95 WatchSource:0}: Error finding container c797ddbd7447e7056e6ec2a34c981ab16b3b09f9f47e4b1cf6b09c5cf95b4e95: Status 404 returned error can't find the container with id c797ddbd7447e7056e6ec2a34c981ab16b3b09f9f47e4b1cf6b09c5cf95b4e95
Sep 18 21:01:54.732045 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.731994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" event={"ID":"015b3fcf-8ede-44c2-9290-2ee095f1af0c","Type":"ContainerStarted","Data":"1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3"}
Sep 18 21:01:54.732200 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:54.732065    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" event={"ID":"015b3fcf-8ede-44c2-9290-2ee095f1af0c","Type":"ContainerStarted","Data":"c797ddbd7447e7056e6ec2a34c981ab16b3b09f9f47e4b1cf6b09c5cf95b4e95"}
Sep 18 21:01:58.445165 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:58.445125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:01:58.749563 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:58.749487    2566 generic.go:356] "Generic (PLEG): container finished" podID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerID="1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3" exitCode=0
Sep 18 21:01:58.749691 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:58.749559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" event={"ID":"015b3fcf-8ede-44c2-9290-2ee095f1af0c","Type":"ContainerDied","Data":"1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3"}
Sep 18 21:01:59.755454 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:59.755409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" event={"ID":"015b3fcf-8ede-44c2-9290-2ee095f1af0c","Type":"ContainerStarted","Data":"44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989"}
Sep 18 21:01:59.755921 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:59.755484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" event={"ID":"015b3fcf-8ede-44c2-9290-2ee095f1af0c","Type":"ContainerStarted","Data":"69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c"}
Sep 18 21:01:59.755921 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:59.755698    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:01:59.774798 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:01:59.774749    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podStartSLOduration=1.863592615 podStartE2EDuration="5.774735368s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.911142753s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:01:59.773815886 +0000 UTC m=+2488.804742026" watchObservedRunningTime="2026-09-18 21:01:59.774735368 +0000 UTC m=+2488.805661508"
Sep 18 21:02:00.759141 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:00.759112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:02:00.760422 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:00.760395    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 18 21:02:01.763522 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:01.763464    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 18 21:02:06.769137 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:06.769106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:02:06.769691 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:06.769658    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 18 21:02:08.444719 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:08.444684    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 18 21:02:14.456205 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:14.456170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:02:16.770064 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:16.769999    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 18 21:02:16.884258 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:16.884223    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"]
Sep 18 21:02:16.884454 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:16.884429    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container" containerID="cri-o://dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0" gracePeriod=30
Sep 18 21:02:16.884532 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:16.884478    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kube-rbac-proxy" containerID="cri-o://cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5" gracePeriod=30
Sep 18 21:02:17.829647 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:17.829615    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerID="cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5" exitCode=2
Sep 18 21:02:17.830034 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:17.829685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" event={"ID":"f4ce29a2-a52d-4eef-852b-cf95d02c3432","Type":"ContainerDied","Data":"cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5"}
Sep 18 21:02:18.439971 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:18.439928    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 18 21:02:22.041921 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.041897    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:02:22.126409 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.126337    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ce29a2-a52d-4eef-852b-cf95d02c3432-proxy-tls\") pod \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") "
Sep 18 21:02:22.126409 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.126385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kserve-provision-location\") pod \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") "
Sep 18 21:02:22.126632 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.126436    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-rbac-proxy-sar-config\") pod \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") "
Sep 18 21:02:22.126632 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.126491    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx9qb\" (UniqueName: \"kubernetes.io/projected/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-api-access-rx9qb\") pod \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") "
Sep 18 21:02:22.126632 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.126537    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-cabundle-cert\") pod \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\" (UID: \"f4ce29a2-a52d-4eef-852b-cf95d02c3432\") "
Sep 18 21:02:22.126805 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.126774    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4ce29a2-a52d-4eef-852b-cf95d02c3432" (UID: "f4ce29a2-a52d-4eef-852b-cf95d02c3432"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:02:22.126903 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.126811    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4ce29a2-a52d-4eef-852b-cf95d02c3432" (UID: "f4ce29a2-a52d-4eef-852b-cf95d02c3432"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:02:22.126960 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.126944    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4ce29a2-a52d-4eef-852b-cf95d02c3432" (UID: "f4ce29a2-a52d-4eef-852b-cf95d02c3432"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:02:22.128497 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.128477    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ce29a2-a52d-4eef-852b-cf95d02c3432-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4ce29a2-a52d-4eef-852b-cf95d02c3432" (UID: "f4ce29a2-a52d-4eef-852b-cf95d02c3432"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:02:22.128597 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.128527    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-api-access-rx9qb" (OuterVolumeSpecName: "kube-api-access-rx9qb") pod "f4ce29a2-a52d-4eef-852b-cf95d02c3432" (UID: "f4ce29a2-a52d-4eef-852b-cf95d02c3432"). InnerVolumeSpecName "kube-api-access-rx9qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:02:22.227621 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.227596    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx9qb\" (UniqueName: \"kubernetes.io/projected/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-api-access-rx9qb\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:02:22.227741 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.227618    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:02:22.227741 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.227636    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4ce29a2-a52d-4eef-852b-cf95d02c3432-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:02:22.227741 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.227644    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:02:22.227741 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.227653    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4ce29a2-a52d-4eef-852b-cf95d02c3432-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:02:22.849249 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.849212    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerID="dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0" exitCode=0
Sep 18 21:02:22.849421 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.849295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" event={"ID":"f4ce29a2-a52d-4eef-852b-cf95d02c3432","Type":"ContainerDied","Data":"dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0"}
Sep 18 21:02:22.849421 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.849301    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"
Sep 18 21:02:22.849421 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.849394    2566 scope.go:117] "RemoveContainer" containerID="cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5"
Sep 18 21:02:22.849561 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.849381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56" event={"ID":"f4ce29a2-a52d-4eef-852b-cf95d02c3432","Type":"ContainerDied","Data":"fb88495e81cec17c537cacba608ced3a6c3ff1561fefec8c06c8b19af2840fab"}
Sep 18 21:02:22.858293 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.858274    2566 scope.go:117] "RemoveContainer" containerID="dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0"
Sep 18 21:02:22.866473 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.866447    2566 scope.go:117] "RemoveContainer" containerID="9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b"
Sep 18 21:02:22.872203 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.872178    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"]
Sep 18 21:02:22.874605 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.874586    2566 scope.go:117] "RemoveContainer" containerID="cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5"
Sep 18 21:02:22.874979 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:02:22.874955    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5\": container with ID starting with cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5 not found: ID does not exist" containerID="cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5"
Sep 18 21:02:22.875100 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.874986    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5"} err="failed to get container status \"cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5\": rpc error: code = NotFound desc = could not find container \"cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5\": container with ID starting with cb65ae663a1a283a2f0be3869bac2e108051b78ea198ba7691c2e3b7685554e5 not found: ID does not exist"
Sep 18 21:02:22.875100 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.875005    2566 scope.go:117] "RemoveContainer" containerID="dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0"
Sep 18 21:02:22.875652 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:02:22.875355    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0\": container with ID starting with dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0 not found: ID does not exist" containerID="dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0"
Sep 18 21:02:22.875862 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.875830    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0"} err="failed to get container status \"dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0\": rpc error: code = NotFound desc = could not find container \"dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0\": container with ID starting with dfb5a0c57153758a2698d04cf7edbb8147be5d3ab60c580d0ebb4ff03b9fcae0 not found: ID does not exist"
Sep 18 21:02:22.875930 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.875866    2566 scope.go:117] "RemoveContainer" containerID="9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b"
Sep 18 21:02:22.876247 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:02:22.876222    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b\": container with ID starting with 9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b not found: ID does not exist" containerID="9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b"
Sep 18 21:02:22.876320 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.876257    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b"} err="failed to get container status \"9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b\": rpc error: code = NotFound desc = could not find container \"9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b\": container with ID starting with 9746668b49fb5ac5df458caa8befe8ecb8f9b838a063cb5370b641c381128f6b not found: ID does not exist"
Sep 18 21:02:22.877248 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:22.877232    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-zqc56"]
Sep 18 21:02:23.459167 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:23.459137    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" path="/var/lib/kubelet/pods/f4ce29a2-a52d-4eef-852b-cf95d02c3432/volumes"
Sep 18 21:02:26.770411 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:26.770311    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 18 21:02:36.769888 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:36.769848    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 18 21:02:46.769701 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:46.769653    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 18 21:02:56.770436 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:02:56.770394    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 18 21:03:06.770716 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:06.770688    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:03:14.702163 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:14.702131    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"]
Sep 18 21:03:14.702678 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:14.702346    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" containerID="cri-o://69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c" gracePeriod=30
Sep 18 21:03:14.702678 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:14.702399    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kube-rbac-proxy" containerID="cri-o://44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989" gracePeriod=30
Sep 18 21:03:15.049182 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:15.049152    2566 generic.go:356] "Generic (PLEG): container finished" podID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerID="44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989" exitCode=2
Sep 18 21:03:15.049339 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:15.049190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" event={"ID":"015b3fcf-8ede-44c2-9290-2ee095f1af0c","Type":"ContainerDied","Data":"44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989"}
Sep 18 21:03:16.764283 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:16.764236    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 18 21:03:16.769661 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:16.769637    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 18 21:03:17.562304 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.562273    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"]
Sep 18 21:03:17.562559 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.562548    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kube-rbac-proxy"
Sep 18 21:03:17.562607 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.562560    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kube-rbac-proxy"
Sep 18 21:03:17.562607 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.562569    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container"
Sep 18 21:03:17.562607 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.562575    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container"
Sep 18 21:03:17.562607 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.562585    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="storage-initializer"
Sep 18 21:03:17.562607 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.562592    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="storage-initializer"
Sep 18 21:03:17.562767 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.562645    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kserve-container"
Sep 18 21:03:17.562767 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.562655    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ce29a2-a52d-4eef-852b-cf95d02c3432" containerName="kube-rbac-proxy"
Sep 18 21:03:17.565706 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.565691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.572601 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.572575    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 18 21:03:17.572739 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.572609    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 21:03:17.591299 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.591271    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"]
Sep 18 21:03:17.670827 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.670794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d37756-81d5-41d1-bf25-162ee3b20cf3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.670980 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.670860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.670980 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.670890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.670980 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.670916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d37756-81d5-41d1-bf25-162ee3b20cf3-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.671161 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.671064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lllk\" (UniqueName: \"kubernetes.io/projected/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-api-access-2lllk\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.772415 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.772386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d37756-81d5-41d1-bf25-162ee3b20cf3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.772866 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.772454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.772866 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.772486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.772866 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.772513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d37756-81d5-41d1-bf25-162ee3b20cf3-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.772866 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:17.772542    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 18 21:03:17.772866 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.772580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lllk\" (UniqueName: \"kubernetes.io/projected/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-api-access-2lllk\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.773157 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:17.772616    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d37756-81d5-41d1-bf25-162ee3b20cf3-proxy-tls podName:14d37756-81d5-41d1-bf25-162ee3b20cf3 nodeName:}" failed. No retries permitted until 2026-09-18 21:03:18.272595158 +0000 UTC m=+2567.303521291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/14d37756-81d5-41d1-bf25-162ee3b20cf3-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" (UID: "14d37756-81d5-41d1-bf25-162ee3b20cf3") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 18 21:03:17.773157 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.773073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d37756-81d5-41d1-bf25-162ee3b20cf3-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.773279 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.773260    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.773316 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.773273    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:17.781090 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:17.781069    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lllk\" (UniqueName: \"kubernetes.io/projected/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-api-access-2lllk\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:18.277485 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.277447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d37756-81d5-41d1-bf25-162ee3b20cf3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:18.279893 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.279864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d37756-81d5-41d1-bf25-162ee3b20cf3-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:18.477097 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.477063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:18.724001 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.723978    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:03:18.748068 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.748044    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"]
Sep 18 21:03:18.750758 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:03:18.750736    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d37756_81d5_41d1_bf25_162ee3b20cf3.slice/crio-14fe8036fb693585aabdef6b19e8c3a2e672b1211b156c94cc0ea41730c6463b WatchSource:0}: Error finding container 14fe8036fb693585aabdef6b19e8c3a2e672b1211b156c94cc0ea41730c6463b: Status 404 returned error can't find the container with id 14fe8036fb693585aabdef6b19e8c3a2e672b1211b156c94cc0ea41730c6463b
Sep 18 21:03:18.782334 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.782285    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-cabundle-cert\") pod \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") "
Sep 18 21:03:18.782334 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.782326    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kserve-provision-location\") pod \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") "
Sep 18 21:03:18.782748 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.782363    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-rbac-proxy-sar-config\") pod \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") "
Sep 18 21:03:18.782748 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.782466    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmj95\" (UniqueName: \"kubernetes.io/projected/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-api-access-zmj95\") pod \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") "
Sep 18 21:03:18.782748 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.782514    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015b3fcf-8ede-44c2-9290-2ee095f1af0c-proxy-tls\") pod \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\" (UID: \"015b3fcf-8ede-44c2-9290-2ee095f1af0c\") "
Sep 18 21:03:18.782748 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.782642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "015b3fcf-8ede-44c2-9290-2ee095f1af0c" (UID: "015b3fcf-8ede-44c2-9290-2ee095f1af0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:03:18.782748 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.782678    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "015b3fcf-8ede-44c2-9290-2ee095f1af0c" (UID: "015b3fcf-8ede-44c2-9290-2ee095f1af0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:03:18.782748 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.782699    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "015b3fcf-8ede-44c2-9290-2ee095f1af0c" (UID: "015b3fcf-8ede-44c2-9290-2ee095f1af0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:03:18.782748 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.782718    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:18.784406 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.784385    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-api-access-zmj95" (OuterVolumeSpecName: "kube-api-access-zmj95") pod "015b3fcf-8ede-44c2-9290-2ee095f1af0c" (UID: "015b3fcf-8ede-44c2-9290-2ee095f1af0c"). InnerVolumeSpecName "kube-api-access-zmj95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:03:18.784469 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.784385    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015b3fcf-8ede-44c2-9290-2ee095f1af0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "015b3fcf-8ede-44c2-9290-2ee095f1af0c" (UID: "015b3fcf-8ede-44c2-9290-2ee095f1af0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:03:18.883828 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.883798    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:18.883828 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.883821    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmj95\" (UniqueName: \"kubernetes.io/projected/015b3fcf-8ede-44c2-9290-2ee095f1af0c-kube-api-access-zmj95\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:18.883828 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.883832    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/015b3fcf-8ede-44c2-9290-2ee095f1af0c-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:18.884053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:18.883841    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/015b3fcf-8ede-44c2-9290-2ee095f1af0c-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:19.068362 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.068275    2566 generic.go:356] "Generic (PLEG): container finished" podID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerID="69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c" exitCode=0
Sep 18 21:03:19.068362 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.068334    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"
Sep 18 21:03:19.068362 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.068339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" event={"ID":"015b3fcf-8ede-44c2-9290-2ee095f1af0c","Type":"ContainerDied","Data":"69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c"}
Sep 18 21:03:19.068649 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.068415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w" event={"ID":"015b3fcf-8ede-44c2-9290-2ee095f1af0c","Type":"ContainerDied","Data":"c797ddbd7447e7056e6ec2a34c981ab16b3b09f9f47e4b1cf6b09c5cf95b4e95"}
Sep 18 21:03:19.068649 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.068452    2566 scope.go:117] "RemoveContainer" containerID="44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989"
Sep 18 21:03:19.070314 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.070289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" event={"ID":"14d37756-81d5-41d1-bf25-162ee3b20cf3","Type":"ContainerStarted","Data":"219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279"}
Sep 18 21:03:19.070454 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.070338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" event={"ID":"14d37756-81d5-41d1-bf25-162ee3b20cf3","Type":"ContainerStarted","Data":"14fe8036fb693585aabdef6b19e8c3a2e672b1211b156c94cc0ea41730c6463b"}
Sep 18 21:03:19.079471 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.079372    2566 scope.go:117] "RemoveContainer" containerID="69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c"
Sep 18 21:03:19.087591 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.087575    2566 scope.go:117] "RemoveContainer" containerID="1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3"
Sep 18 21:03:19.095109 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.095047    2566 scope.go:117] "RemoveContainer" containerID="44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989"
Sep 18 21:03:19.095330 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:19.095312    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989\": container with ID starting with 44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989 not found: ID does not exist" containerID="44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989"
Sep 18 21:03:19.095405 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.095336    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989"} err="failed to get container status \"44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989\": rpc error: code = NotFound desc = could not find container \"44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989\": container with ID starting with 44d1d911f942c16d40c5855d04a95de02afb43c3253900c65b67776cc186b989 not found: ID does not exist"
Sep 18 21:03:19.095405 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.095352    2566 scope.go:117] "RemoveContainer" containerID="69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c"
Sep 18 21:03:19.095597 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:19.095577    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c\": container with ID starting with 69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c not found: ID does not exist" containerID="69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c"
Sep 18 21:03:19.095658 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.095603    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c"} err="failed to get container status \"69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c\": rpc error: code = NotFound desc = could not find container \"69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c\": container with ID starting with 69eaf9a119df1bf75bc5cb05fc6d6e922afb3480d5a332910a559d6f9c28862c not found: ID does not exist"
Sep 18 21:03:19.095658 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.095618    2566 scope.go:117] "RemoveContainer" containerID="1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3"
Sep 18 21:03:19.095862 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:19.095846    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3\": container with ID starting with 1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3 not found: ID does not exist" containerID="1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3"
Sep 18 21:03:19.095926 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.095865    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3"} err="failed to get container status \"1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3\": rpc error: code = NotFound desc = could not find container \"1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3\": container with ID starting with 1af017f7b452ef8a87c809110a261aa7ac85f74314692b1e6eabb0e369860cf3 not found: ID does not exist"
Sep 18 21:03:19.103851 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.103830    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"]
Sep 18 21:03:19.107651 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.107629    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-5db4w"]
Sep 18 21:03:19.465285 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.460848    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" path="/var/lib/kubelet/pods/015b3fcf-8ede-44c2-9290-2ee095f1af0c/volumes"
Sep 18 21:03:19.898466 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.898430    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"]
Sep 18 21:03:19.898856 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.898842    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container"
Sep 18 21:03:19.898901 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.898859    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container"
Sep 18 21:03:19.898901 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.898873    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="storage-initializer"
Sep 18 21:03:19.898901 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.898881    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="storage-initializer"
Sep 18 21:03:19.898992 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.898915    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kube-rbac-proxy"
Sep 18 21:03:19.898992 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.898923    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kube-rbac-proxy"
Sep 18 21:03:19.899084 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.898999    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kserve-container"
Sep 18 21:03:19.899084 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.899025    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="015b3fcf-8ede-44c2-9290-2ee095f1af0c" containerName="kube-rbac-proxy"
Sep 18 21:03:19.903333 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.903316    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:19.905689 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.905666    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 18 21:03:19.905822 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.905731    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 18 21:03:19.905822 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.905805    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 21:03:19.911929 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.911904    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"]
Sep 18 21:03:19.993408 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.993362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e244118-3dc9-4e6b-84b3-ea5a46557823-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:19.993555 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.993421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkh6w\" (UniqueName: \"kubernetes.io/projected/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-api-access-wkh6w\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:19.993555 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.993488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e244118-3dc9-4e6b-84b3-ea5a46557823-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:19.993555 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:19.993537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.093858 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.093833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.094050 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.093872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e244118-3dc9-4e6b-84b3-ea5a46557823-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.094050 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.093901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkh6w\" (UniqueName: \"kubernetes.io/projected/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-api-access-wkh6w\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.094050 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.093937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e244118-3dc9-4e6b-84b3-ea5a46557823-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.094050 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:20.093992    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 18 21:03:20.094261 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:20.094074    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e244118-3dc9-4e6b-84b3-ea5a46557823-proxy-tls podName:0e244118-3dc9-4e6b-84b3-ea5a46557823 nodeName:}" failed. No retries permitted until 2026-09-18 21:03:20.594054779 +0000 UTC m=+2569.624980898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0e244118-3dc9-4e6b-84b3-ea5a46557823-proxy-tls") pod "isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" (UID: "0e244118-3dc9-4e6b-84b3-ea5a46557823") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 18 21:03:20.094326 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.094302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e244118-3dc9-4e6b-84b3-ea5a46557823-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.094658 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.094636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.102972 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.102940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkh6w\" (UniqueName: \"kubernetes.io/projected/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-api-access-wkh6w\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.599029 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.598985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e244118-3dc9-4e6b-84b3-ea5a46557823-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.601353 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.601325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e244118-3dc9-4e6b-84b3-ea5a46557823-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-hw6jt\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:20.813511 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:20.813477    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:21.085397 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:21.085374    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"]
Sep 18 21:03:21.086942 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:03:21.086913    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e244118_3dc9_4e6b_84b3_ea5a46557823.slice/crio-63c3d71af48a0796e5266164e7fc6e9e2ebf43b505e8c931e38d14b59e10a5e7 WatchSource:0}: Error finding container 63c3d71af48a0796e5266164e7fc6e9e2ebf43b505e8c931e38d14b59e10a5e7: Status 404 returned error can't find the container with id 63c3d71af48a0796e5266164e7fc6e9e2ebf43b505e8c931e38d14b59e10a5e7
Sep 18 21:03:22.084663 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:22.084629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" event={"ID":"0e244118-3dc9-4e6b-84b3-ea5a46557823","Type":"ContainerStarted","Data":"e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7"}
Sep 18 21:03:22.084851 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:22.084685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" event={"ID":"0e244118-3dc9-4e6b-84b3-ea5a46557823","Type":"ContainerStarted","Data":"63c3d71af48a0796e5266164e7fc6e9e2ebf43b505e8c931e38d14b59e10a5e7"}
Sep 18 21:03:23.089618 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:23.089536    2566 generic.go:356] "Generic (PLEG): container finished" podID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerID="219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279" exitCode=0
Sep 18 21:03:23.090052 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:23.089609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" event={"ID":"14d37756-81d5-41d1-bf25-162ee3b20cf3","Type":"ContainerDied","Data":"219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279"}
Sep 18 21:03:23.091218 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:23.091189    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerID="e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7" exitCode=0
Sep 18 21:03:23.091351 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:23.091217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" event={"ID":"0e244118-3dc9-4e6b-84b3-ea5a46557823","Type":"ContainerDied","Data":"e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7"}
Sep 18 21:03:24.097664 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.097627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" event={"ID":"14d37756-81d5-41d1-bf25-162ee3b20cf3","Type":"ContainerStarted","Data":"e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8"}
Sep 18 21:03:24.098119 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.097710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" event={"ID":"14d37756-81d5-41d1-bf25-162ee3b20cf3","Type":"ContainerStarted","Data":"6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2"}
Sep 18 21:03:24.098119 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.098047    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:24.098218 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.098198    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:24.101384 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.101349    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 18 21:03:24.103146 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.103122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" event={"ID":"0e244118-3dc9-4e6b-84b3-ea5a46557823","Type":"ContainerStarted","Data":"0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111"}
Sep 18 21:03:24.103247 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.103170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" event={"ID":"0e244118-3dc9-4e6b-84b3-ea5a46557823","Type":"ContainerStarted","Data":"4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a"}
Sep 18 21:03:24.103247 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.103242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:24.121526 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.121478    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" podStartSLOduration=3.166246787 podStartE2EDuration="7.121464273s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.955217486s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:03:24.120554768 +0000 UTC m=+2573.151480988" watchObservedRunningTime="2026-09-18 21:03:24.121464273 +0000 UTC m=+2573.152390412"
Sep 18 21:03:24.139682 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:24.139640    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podStartSLOduration=4.114804999 podStartE2EDuration="5.139626051s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.024821052s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:03:24.137899308 +0000 UTC m=+2573.168825463" watchObservedRunningTime="2026-09-18 21:03:24.139626051 +0000 UTC m=+2573.170552193"
Sep 18 21:03:25.107338 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:25.107305    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:25.107730 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:25.107305    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 18 21:03:25.108757 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:25.108731    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:03:26.111249 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:26.111205    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:03:30.112144 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:30.112109    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:30.112664 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:30.112639    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 18 21:03:31.116334 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:31.116304    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:03:31.116776 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:31.116749    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:03:40.113808 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:40.113778    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:41.117841 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:41.117796    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:03:48.201567 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:48.201535    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"]
Sep 18 21:03:48.202090 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:48.201755    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kserve-container" containerID="cri-o://6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2" gracePeriod=30
Sep 18 21:03:48.202090 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:48.201809    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kube-rbac-proxy" containerID="cri-o://e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8" gracePeriod=30
Sep 18 21:03:48.951689 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:48.951667    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:49.133764 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.133732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d37756-81d5-41d1-bf25-162ee3b20cf3-proxy-tls\") pod \"14d37756-81d5-41d1-bf25-162ee3b20cf3\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") "
Sep 18 21:03:49.133940 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.133789    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d37756-81d5-41d1-bf25-162ee3b20cf3-kserve-provision-location\") pod \"14d37756-81d5-41d1-bf25-162ee3b20cf3\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") "
Sep 18 21:03:49.133940 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.133815    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lllk\" (UniqueName: \"kubernetes.io/projected/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-api-access-2lllk\") pod \"14d37756-81d5-41d1-bf25-162ee3b20cf3\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") "
Sep 18 21:03:49.133940 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.133872    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-rbac-proxy-sar-config\") pod \"14d37756-81d5-41d1-bf25-162ee3b20cf3\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") "
Sep 18 21:03:49.133940 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.133911    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-cabundle-cert\") pod \"14d37756-81d5-41d1-bf25-162ee3b20cf3\" (UID: \"14d37756-81d5-41d1-bf25-162ee3b20cf3\") "
Sep 18 21:03:49.134236 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.134212    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14d37756-81d5-41d1-bf25-162ee3b20cf3" (UID: "14d37756-81d5-41d1-bf25-162ee3b20cf3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:03:49.134297 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.134236    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d37756-81d5-41d1-bf25-162ee3b20cf3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14d37756-81d5-41d1-bf25-162ee3b20cf3" (UID: "14d37756-81d5-41d1-bf25-162ee3b20cf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:03:49.134337 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.134298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14d37756-81d5-41d1-bf25-162ee3b20cf3" (UID: "14d37756-81d5-41d1-bf25-162ee3b20cf3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:03:49.135857 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.135838    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-api-access-2lllk" (OuterVolumeSpecName: "kube-api-access-2lllk") pod "14d37756-81d5-41d1-bf25-162ee3b20cf3" (UID: "14d37756-81d5-41d1-bf25-162ee3b20cf3"). InnerVolumeSpecName "kube-api-access-2lllk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:03:49.135939 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.135864    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d37756-81d5-41d1-bf25-162ee3b20cf3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14d37756-81d5-41d1-bf25-162ee3b20cf3" (UID: "14d37756-81d5-41d1-bf25-162ee3b20cf3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:03:49.204145 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.204118    2566 generic.go:356] "Generic (PLEG): container finished" podID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerID="e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8" exitCode=2
Sep 18 21:03:49.204145 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.204139    2566 generic.go:356] "Generic (PLEG): container finished" podID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerID="6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2" exitCode=0
Sep 18 21:03:49.204577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.204170    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"
Sep 18 21:03:49.204577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.204230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" event={"ID":"14d37756-81d5-41d1-bf25-162ee3b20cf3","Type":"ContainerDied","Data":"e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8"}
Sep 18 21:03:49.204577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.204281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" event={"ID":"14d37756-81d5-41d1-bf25-162ee3b20cf3","Type":"ContainerDied","Data":"6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2"}
Sep 18 21:03:49.204577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.204302    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h" event={"ID":"14d37756-81d5-41d1-bf25-162ee3b20cf3","Type":"ContainerDied","Data":"14fe8036fb693585aabdef6b19e8c3a2e672b1211b156c94cc0ea41730c6463b"}
Sep 18 21:03:49.204577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.204326    2566 scope.go:117] "RemoveContainer" containerID="e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8"
Sep 18 21:03:49.213255 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.213230    2566 scope.go:117] "RemoveContainer" containerID="6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2"
Sep 18 21:03:49.220675 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.220658    2566 scope.go:117] "RemoveContainer" containerID="219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279"
Sep 18 21:03:49.226842 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.226820    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"]
Sep 18 21:03:49.228992 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.228975    2566 scope.go:117] "RemoveContainer" containerID="e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8"
Sep 18 21:03:49.229298 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:49.229276    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8\": container with ID starting with e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8 not found: ID does not exist" containerID="e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8"
Sep 18 21:03:49.229355 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.229310    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8"} err="failed to get container status \"e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8\": rpc error: code = NotFound desc = could not find container \"e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8\": container with ID starting with e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8 not found: ID does not exist"
Sep 18 21:03:49.229355 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.229336    2566 scope.go:117] "RemoveContainer" containerID="6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2"
Sep 18 21:03:49.229622 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:49.229605    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2\": container with ID starting with 6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2 not found: ID does not exist" containerID="6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2"
Sep 18 21:03:49.229663 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.229628    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2"} err="failed to get container status \"6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2\": rpc error: code = NotFound desc = could not find container \"6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2\": container with ID starting with 6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2 not found: ID does not exist"
Sep 18 21:03:49.229663 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.229646    2566 scope.go:117] "RemoveContainer" containerID="219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279"
Sep 18 21:03:49.229893 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:03:49.229864    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279\": container with ID starting with 219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279 not found: ID does not exist" containerID="219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279"
Sep 18 21:03:49.229978 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.229890    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279"} err="failed to get container status \"219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279\": rpc error: code = NotFound desc = could not find container \"219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279\": container with ID starting with 219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279 not found: ID does not exist"
Sep 18 21:03:49.229978 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.229906    2566 scope.go:117] "RemoveContainer" containerID="e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8"
Sep 18 21:03:49.230242 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.230218    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8"} err="failed to get container status \"e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8\": rpc error: code = NotFound desc = could not find container \"e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8\": container with ID starting with e7784fd1e777c261c010af804483e281e985b9e44c9f2db757d5534f9a1f18a8 not found: ID does not exist"
Sep 18 21:03:49.230298 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.230246    2566 scope.go:117] "RemoveContainer" containerID="6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2"
Sep 18 21:03:49.230385 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.230369    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cn22h"]
Sep 18 21:03:49.230512 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.230492    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2"} err="failed to get container status \"6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2\": rpc error: code = NotFound desc = could not find container \"6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2\": container with ID starting with 6684a555bfbe118d1bc3329d02229ed96a978e281c1f54524eb68cc99a127ea2 not found: ID does not exist"
Sep 18 21:03:49.230586 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.230515    2566 scope.go:117] "RemoveContainer" containerID="219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279"
Sep 18 21:03:49.230769 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.230752    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279"} err="failed to get container status \"219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279\": rpc error: code = NotFound desc = could not find container \"219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279\": container with ID starting with 219724252ed3799b5d202360307f9f7fd7c9d3fd19b961b5fd160daeca973279 not found: ID does not exist"
Sep 18 21:03:49.235182 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.235160    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d37756-81d5-41d1-bf25-162ee3b20cf3-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:49.235262 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.235185    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lllk\" (UniqueName: \"kubernetes.io/projected/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-api-access-2lllk\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:49.235262 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.235199    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:49.235262 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.235211    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d37756-81d5-41d1-bf25-162ee3b20cf3-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:49.235262 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.235224    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d37756-81d5-41d1-bf25-162ee3b20cf3-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:03:49.459591 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:49.459498    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" path="/var/lib/kubelet/pods/14d37756-81d5-41d1-bf25-162ee3b20cf3/volumes"
Sep 18 21:03:51.117510 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:03:51.117472    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:04:01.117659 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:01.117580    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:04:11.117684 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:11.117642    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:04:21.117185 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:21.117143    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:04:31.116935 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:31.116897    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:04:41.117716 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:41.117686    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:04:50.491424 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.491380    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"]
Sep 18 21:04:50.491938 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.491738    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" containerID="cri-o://4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a" gracePeriod=30
Sep 18 21:04:50.491938 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.491768    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kube-rbac-proxy" containerID="cri-o://0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111" gracePeriod=30
Sep 18 21:04:50.700424 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.700391    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"]
Sep 18 21:04:50.700701 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.700689    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="storage-initializer"
Sep 18 21:04:50.700750 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.700703    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="storage-initializer"
Sep 18 21:04:50.700750 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.700714    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kube-rbac-proxy"
Sep 18 21:04:50.700750 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.700719    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kube-rbac-proxy"
Sep 18 21:04:50.700750 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.700731    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kserve-container"
Sep 18 21:04:50.700750 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.700737    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kserve-container"
Sep 18 21:04:50.700910 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.700783    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kserve-container"
Sep 18 21:04:50.700910 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.700792    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d37756-81d5-41d1-bf25-162ee3b20cf3" containerName="kube-rbac-proxy"
Sep 18 21:04:50.703916 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.703901    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.706581 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.706563    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 18 21:04:50.706773 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.706750    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 18 21:04:50.706941 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.706927    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 21:04:50.714748 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.714725    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"]
Sep 18 21:04:50.766305 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.766274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjjf\" (UniqueName: \"kubernetes.io/projected/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-api-access-fzjjf\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.766430 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.766316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.766430 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.766372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.766544 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.766463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.766544 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.766489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.867102 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.867069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.867234 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.867114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.867234 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.867155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjjf\" (UniqueName: \"kubernetes.io/projected/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-api-access-fzjjf\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.867234 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.867177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.867234 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.867203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.867678 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.867650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.867874 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.867857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.868053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.868004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.869578 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.869549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:50.878366 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:50.878347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjjf\" (UniqueName: \"kubernetes.io/projected/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-api-access-fzjjf\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:51.014872 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:51.014835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:51.111846 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:51.111809    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Sep 18 21:04:51.117173 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:51.117150    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 18 21:04:51.285905 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:51.285860    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"]
Sep 18 21:04:51.288201 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:04:51.288174    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcdff2c_e9e2_4bde_ae13_b6c5750c9fb7.slice/crio-8a83cdd9a3c74ddfd98a8127218e9adbd5ce538984b36bbe46535db76b5a3b7d WatchSource:0}: Error finding container 8a83cdd9a3c74ddfd98a8127218e9adbd5ce538984b36bbe46535db76b5a3b7d: Status 404 returned error can't find the container with id 8a83cdd9a3c74ddfd98a8127218e9adbd5ce538984b36bbe46535db76b5a3b7d
Sep 18 21:04:51.457119 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:51.457084    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerID="0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111" exitCode=2
Sep 18 21:04:51.459815 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:51.459782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" event={"ID":"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7","Type":"ContainerStarted","Data":"a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1"}
Sep 18 21:04:51.459929 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:51.459852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" event={"ID":"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7","Type":"ContainerStarted","Data":"8a83cdd9a3c74ddfd98a8127218e9adbd5ce538984b36bbe46535db76b5a3b7d"}
Sep 18 21:04:51.459929 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:51.459884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" event={"ID":"0e244118-3dc9-4e6b-84b3-ea5a46557823","Type":"ContainerDied","Data":"0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111"}
Sep 18 21:04:52.462360 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:52.462330    2566 generic.go:356] "Generic (PLEG): container finished" podID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerID="a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1" exitCode=0
Sep 18 21:04:52.462721 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:52.462406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" event={"ID":"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7","Type":"ContainerDied","Data":"a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1"}
Sep 18 21:04:53.468192 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:53.468153    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" event={"ID":"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7","Type":"ContainerStarted","Data":"abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9"}
Sep 18 21:04:53.468649 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:53.468212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" event={"ID":"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7","Type":"ContainerStarted","Data":"197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e"}
Sep 18 21:04:53.468649 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:53.468425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:53.468649 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:53.468483    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:53.470044 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:53.469992    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 18 21:04:53.488236 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:53.488187    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podStartSLOduration=2.447314037 podStartE2EDuration="3.488173256s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.040859219s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:04:53.486273018 +0000 UTC m=+2662.517199157" watchObservedRunningTime="2026-09-18 21:04:53.488173256 +0000 UTC m=+2662.519099397"
Sep 18 21:04:54.472816 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:54.472777    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 18 21:04:55.333176 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.333154    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:04:55.405030 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.404987    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-rbac-proxy-sar-config\") pod \"0e244118-3dc9-4e6b-84b3-ea5a46557823\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") "
Sep 18 21:04:55.405183 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.405054    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e244118-3dc9-4e6b-84b3-ea5a46557823-proxy-tls\") pod \"0e244118-3dc9-4e6b-84b3-ea5a46557823\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") "
Sep 18 21:04:55.405183 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.405111    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkh6w\" (UniqueName: \"kubernetes.io/projected/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-api-access-wkh6w\") pod \"0e244118-3dc9-4e6b-84b3-ea5a46557823\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") "
Sep 18 21:04:55.405183 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.405140    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e244118-3dc9-4e6b-84b3-ea5a46557823-kserve-provision-location\") pod \"0e244118-3dc9-4e6b-84b3-ea5a46557823\" (UID: \"0e244118-3dc9-4e6b-84b3-ea5a46557823\") "
Sep 18 21:04:55.405446 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.405422    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e244118-3dc9-4e6b-84b3-ea5a46557823" (UID: "0e244118-3dc9-4e6b-84b3-ea5a46557823"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:04:55.405537 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.405511    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e244118-3dc9-4e6b-84b3-ea5a46557823-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e244118-3dc9-4e6b-84b3-ea5a46557823" (UID: "0e244118-3dc9-4e6b-84b3-ea5a46557823"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:04:55.407200 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.407176    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e244118-3dc9-4e6b-84b3-ea5a46557823-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e244118-3dc9-4e6b-84b3-ea5a46557823" (UID: "0e244118-3dc9-4e6b-84b3-ea5a46557823"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:04:55.407268 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.407183    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-api-access-wkh6w" (OuterVolumeSpecName: "kube-api-access-wkh6w") pod "0e244118-3dc9-4e6b-84b3-ea5a46557823" (UID: "0e244118-3dc9-4e6b-84b3-ea5a46557823"). InnerVolumeSpecName "kube-api-access-wkh6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:04:55.477723 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.477692    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerID="4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a" exitCode=0
Sep 18 21:04:55.478042 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.477745    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" event={"ID":"0e244118-3dc9-4e6b-84b3-ea5a46557823","Type":"ContainerDied","Data":"4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a"}
Sep 18 21:04:55.478042 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.477789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt" event={"ID":"0e244118-3dc9-4e6b-84b3-ea5a46557823","Type":"ContainerDied","Data":"63c3d71af48a0796e5266164e7fc6e9e2ebf43b505e8c931e38d14b59e10a5e7"}
Sep 18 21:04:55.478042 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.477816    2566 scope.go:117] "RemoveContainer" containerID="0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111"
Sep 18 21:04:55.478042 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.477751    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"
Sep 18 21:04:55.485764 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.485739    2566 scope.go:117] "RemoveContainer" containerID="4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a"
Sep 18 21:04:55.492784 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.492764    2566 scope.go:117] "RemoveContainer" containerID="e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7"
Sep 18 21:04:55.495682 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.495661    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"]
Sep 18 21:04:55.499903 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.499882    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-hw6jt"]
Sep 18 21:04:55.501229 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.501211    2566 scope.go:117] "RemoveContainer" containerID="0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111"
Sep 18 21:04:55.501504 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:04:55.501487    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111\": container with ID starting with 0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111 not found: ID does not exist" containerID="0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111"
Sep 18 21:04:55.501592 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.501514    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111"} err="failed to get container status \"0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111\": rpc error: code = NotFound desc = could not find container \"0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111\": container with ID starting with 0ff650ee0cd42befe3c19fa40d100a0aa68378fd18482216e3b8b12142ec6111 not found: ID does not exist"
Sep 18 21:04:55.501592 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.501531    2566 scope.go:117] "RemoveContainer" containerID="4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a"
Sep 18 21:04:55.501778 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:04:55.501760    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a\": container with ID starting with 4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a not found: ID does not exist" containerID="4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a"
Sep 18 21:04:55.501848 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.501788    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a"} err="failed to get container status \"4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a\": rpc error: code = NotFound desc = could not find container \"4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a\": container with ID starting with 4175784336bf268c8dab194119e48934d86d67c144eebb440da6fac36c701f3a not found: ID does not exist"
Sep 18 21:04:55.501848 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.501809    2566 scope.go:117] "RemoveContainer" containerID="e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7"
Sep 18 21:04:55.502079 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:04:55.502062    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7\": container with ID starting with e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7 not found: ID does not exist" containerID="e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7"
Sep 18 21:04:55.502133 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.502084    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7"} err="failed to get container status \"e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7\": rpc error: code = NotFound desc = could not find container \"e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7\": container with ID starting with e03349a1645d18558ec7bbec656b77a5312161114925238cfa6b18cfc0796ff7 not found: ID does not exist"
Sep 18 21:04:55.506442 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.506424    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:04:55.506494 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.506444    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e244118-3dc9-4e6b-84b3-ea5a46557823-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:04:55.506494 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.506455    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkh6w\" (UniqueName: \"kubernetes.io/projected/0e244118-3dc9-4e6b-84b3-ea5a46557823-kube-api-access-wkh6w\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:04:55.506494 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:55.506463    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e244118-3dc9-4e6b-84b3-ea5a46557823-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:04:57.466133 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:57.466097    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" path="/var/lib/kubelet/pods/0e244118-3dc9-4e6b-84b3-ea5a46557823/volumes"
Sep 18 21:04:59.476970 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:59.476940    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:04:59.477498 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:04:59.477471    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 18 21:05:09.477637 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:05:09.477601    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 18 21:05:19.477749 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:05:19.477706    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 18 21:05:29.477577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:05:29.477498    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 18 21:05:31.559634 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:05:31.559603    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 21:05:31.564438 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:05:31.564419    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 21:05:39.478084 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:05:39.478047    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 18 21:05:49.477619 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:05:49.477582    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 18 21:05:59.477944 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:05:59.477907    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 18 21:06:09.478158 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:09.478129    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:06:10.714419 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:10.714383    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"]
Sep 18 21:06:10.714844 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:10.714600    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container" containerID="cri-o://197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e" gracePeriod=30
Sep 18 21:06:10.714844 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:10.714668    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kube-rbac-proxy" containerID="cri-o://abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9" gracePeriod=30
Sep 18 21:06:11.766139 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.766107    2566 generic.go:356] "Generic (PLEG): container finished" podID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerID="abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9" exitCode=2
Sep 18 21:06:11.766534 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.766190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" event={"ID":"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7","Type":"ContainerDied","Data":"abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9"}
Sep 18 21:06:11.798771 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.798741    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"]
Sep 18 21:06:11.799073 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.799060    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container"
Sep 18 21:06:11.799123 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.799077    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container"
Sep 18 21:06:11.799123 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.799089    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kube-rbac-proxy"
Sep 18 21:06:11.799123 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.799094    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kube-rbac-proxy"
Sep 18 21:06:11.799123 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.799103    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="storage-initializer"
Sep 18 21:06:11.799123 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.799109    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="storage-initializer"
Sep 18 21:06:11.799290 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.799175    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kube-rbac-proxy"
Sep 18 21:06:11.799290 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.799187    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e244118-3dc9-4e6b-84b3-ea5a46557823" containerName="kserve-container"
Sep 18 21:06:11.802181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.802164    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:11.804669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.804648    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 18 21:06:11.804770 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.804652    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 18 21:06:11.812910 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.812887    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"]
Sep 18 21:06:11.903162 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.903119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj75w\" (UniqueName: \"kubernetes.io/projected/c542399e-b39c-446c-8245-fc810ce8e44e-kube-api-access-sj75w\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:11.903300 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.903177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c542399e-b39c-446c-8245-fc810ce8e44e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:11.903300 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.903231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c542399e-b39c-446c-8245-fc810ce8e44e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:11.903379 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:11.903297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c542399e-b39c-446c-8245-fc810ce8e44e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.004556 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.004525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj75w\" (UniqueName: \"kubernetes.io/projected/c542399e-b39c-446c-8245-fc810ce8e44e-kube-api-access-sj75w\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.004556 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.004560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c542399e-b39c-446c-8245-fc810ce8e44e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.004726 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.004581    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c542399e-b39c-446c-8245-fc810ce8e44e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.004726 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.004616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c542399e-b39c-446c-8245-fc810ce8e44e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.004726 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:06:12.004707    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 18 21:06:12.004877 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:06:12.004785    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c542399e-b39c-446c-8245-fc810ce8e44e-proxy-tls podName:c542399e-b39c-446c-8245-fc810ce8e44e nodeName:}" failed. No retries permitted until 2026-09-18 21:06:12.504766015 +0000 UTC m=+2741.535692146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c542399e-b39c-446c-8245-fc810ce8e44e-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4" (UID: "c542399e-b39c-446c-8245-fc810ce8e44e") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 18 21:06:12.005077 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.005057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c542399e-b39c-446c-8245-fc810ce8e44e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.005247 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.005230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c542399e-b39c-446c-8245-fc810ce8e44e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.014652 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.014629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj75w\" (UniqueName: \"kubernetes.io/projected/c542399e-b39c-446c-8245-fc810ce8e44e-kube-api-access-sj75w\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.508118 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.508084    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c542399e-b39c-446c-8245-fc810ce8e44e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.510331 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.510310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c542399e-b39c-446c-8245-fc810ce8e44e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.712951 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.712913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:12.982623 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:12.982600    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"]
Sep 18 21:06:12.984690 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:06:12.984664    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc542399e_b39c_446c_8245_fc810ce8e44e.slice/crio-9f8ed791a90ef90e45907d9b05cab638a4ba79f0df479702d5358d74230506ea WatchSource:0}: Error finding container 9f8ed791a90ef90e45907d9b05cab638a4ba79f0df479702d5358d74230506ea: Status 404 returned error can't find the container with id 9f8ed791a90ef90e45907d9b05cab638a4ba79f0df479702d5358d74230506ea
Sep 18 21:06:13.777956 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:13.777921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4" event={"ID":"c542399e-b39c-446c-8245-fc810ce8e44e","Type":"ContainerStarted","Data":"0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a"}
Sep 18 21:06:13.778230 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:13.777976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4" event={"ID":"c542399e-b39c-446c-8245-fc810ce8e44e","Type":"ContainerStarted","Data":"9f8ed791a90ef90e45907d9b05cab638a4ba79f0df479702d5358d74230506ea"}
Sep 18 21:06:13.906322 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:13.906290    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"]
Sep 18 21:06:13.909607 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:13.909590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:13.912186 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:13.912159    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 18 21:06:13.912186 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:13.912175    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 18 21:06:13.912364 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:13.912165    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 21:06:13.919902 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:13.919876    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"]
Sep 18 21:06:14.021319 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.021285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f424dc-13ff-4300-84c4-39edb809d5ff-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.021668 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.021341    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.021668 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.021358    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f424dc-13ff-4300-84c4-39edb809d5ff-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.021668 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.021417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzf88\" (UniqueName: \"kubernetes.io/projected/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-api-access-nzf88\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.021668 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.021443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.122716 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.122635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f424dc-13ff-4300-84c4-39edb809d5ff-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.122716 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.122710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.122925 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.122736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f424dc-13ff-4300-84c4-39edb809d5ff-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.122925 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.122781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzf88\" (UniqueName: \"kubernetes.io/projected/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-api-access-nzf88\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.122925 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:06:14.122801    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 18 21:06:14.122925 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.122813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.122925 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:06:14.122869    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f424dc-13ff-4300-84c4-39edb809d5ff-proxy-tls podName:a4f424dc-13ff-4300-84c4-39edb809d5ff nodeName:}" failed. No retries permitted until 2026-09-18 21:06:14.622846365 +0000 UTC m=+2743.653772485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a4f424dc-13ff-4300-84c4-39edb809d5ff-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-lj5kw" (UID: "a4f424dc-13ff-4300-84c4-39edb809d5ff") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 18 21:06:14.123279 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.123260    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f424dc-13ff-4300-84c4-39edb809d5ff-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.123431 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.123403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.123522 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.123506    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.133643 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.133615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzf88\" (UniqueName: \"kubernetes.io/projected/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-api-access-nzf88\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.473897 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.473808    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 18 21:06:14.627183 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.627152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f424dc-13ff-4300-84c4-39edb809d5ff-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.629663 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.629639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f424dc-13ff-4300-84c4-39edb809d5ff-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lj5kw\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:14.820685 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:14.820652    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:15.092397 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.092317    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"]
Sep 18 21:06:15.095877 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:06:15.095853    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f424dc_13ff_4300_84c4_39edb809d5ff.slice/crio-930bb55d311deef82f2d9783f42d8cc85e950affbd0109270d0131d16da7a1ee WatchSource:0}: Error finding container 930bb55d311deef82f2d9783f42d8cc85e950affbd0109270d0131d16da7a1ee: Status 404 returned error can't find the container with id 930bb55d311deef82f2d9783f42d8cc85e950affbd0109270d0131d16da7a1ee
Sep 18 21:06:15.554063 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.554039    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:06:15.634519 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.634446    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzjjf\" (UniqueName: \"kubernetes.io/projected/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-api-access-fzjjf\") pod \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") "
Sep 18 21:06:15.634519 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.634482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-rbac-proxy-sar-config\") pod \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") "
Sep 18 21:06:15.634720 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.634550    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-cabundle-cert\") pod \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") "
Sep 18 21:06:15.634720 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.634578    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kserve-provision-location\") pod \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") "
Sep 18 21:06:15.634720 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.634623    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-proxy-tls\") pod \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\" (UID: \"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7\") "
Sep 18 21:06:15.634993 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.634968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" (UID: "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:06:15.634993 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.634976    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" (UID: "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:06:15.635127 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.634984    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" (UID: "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:06:15.636706 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.636681    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-api-access-fzjjf" (OuterVolumeSpecName: "kube-api-access-fzjjf") pod "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" (UID: "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7"). InnerVolumeSpecName "kube-api-access-fzjjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:06:15.636772 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.636710    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" (UID: "bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:06:15.735777 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.735748    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjjf\" (UniqueName: \"kubernetes.io/projected/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-api-access-fzjjf\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:06:15.735777 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.735771    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:06:15.735777 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.735782    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:06:15.735984 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.735790    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:06:15.735984 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.735800    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:06:15.787810 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.787775    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" event={"ID":"a4f424dc-13ff-4300-84c4-39edb809d5ff","Type":"ContainerStarted","Data":"7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674"}
Sep 18 21:06:15.787947 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.787875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" event={"ID":"a4f424dc-13ff-4300-84c4-39edb809d5ff","Type":"ContainerStarted","Data":"930bb55d311deef82f2d9783f42d8cc85e950affbd0109270d0131d16da7a1ee"}
Sep 18 21:06:15.789669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.789645    2566 generic.go:356] "Generic (PLEG): container finished" podID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerID="197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e" exitCode=0
Sep 18 21:06:15.789783 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.789706    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"
Sep 18 21:06:15.789831 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.789710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" event={"ID":"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7","Type":"ContainerDied","Data":"197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e"}
Sep 18 21:06:15.789831 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.789824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh" event={"ID":"bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7","Type":"ContainerDied","Data":"8a83cdd9a3c74ddfd98a8127218e9adbd5ce538984b36bbe46535db76b5a3b7d"}
Sep 18 21:06:15.789904 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.789849    2566 scope.go:117] "RemoveContainer" containerID="abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9"
Sep 18 21:06:15.797893 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.797877    2566 scope.go:117] "RemoveContainer" containerID="197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e"
Sep 18 21:06:15.805675 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.805655    2566 scope.go:117] "RemoveContainer" containerID="a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1"
Sep 18 21:06:15.813467 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.813451    2566 scope.go:117] "RemoveContainer" containerID="abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9"
Sep 18 21:06:15.813730 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:06:15.813712    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9\": container with ID starting with abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9 not found: ID does not exist" containerID="abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9"
Sep 18 21:06:15.813779 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.813738    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9"} err="failed to get container status \"abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9\": rpc error: code = NotFound desc = could not find container \"abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9\": container with ID starting with abb91bb3471ffd1a9db9a99508f005d98ec9252156d1a274f5add09f82259aa9 not found: ID does not exist"
Sep 18 21:06:15.813779 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.813754    2566 scope.go:117] "RemoveContainer" containerID="197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e"
Sep 18 21:06:15.814004 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:06:15.813989    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e\": container with ID starting with 197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e not found: ID does not exist" containerID="197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e"
Sep 18 21:06:15.814004 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.814005    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e"} err="failed to get container status \"197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e\": rpc error: code = NotFound desc = could not find container \"197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e\": container with ID starting with 197b77b193d4d484e126da909dfb929e064e57cec077ae0312a9312a202bc58e not found: ID does not exist"
Sep 18 21:06:15.814117 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.814031    2566 scope.go:117] "RemoveContainer" containerID="a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1"
Sep 18 21:06:15.814284 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:06:15.814270    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1\": container with ID starting with a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1 not found: ID does not exist" containerID="a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1"
Sep 18 21:06:15.814327 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.814286    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1"} err="failed to get container status \"a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1\": rpc error: code = NotFound desc = could not find container \"a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1\": container with ID starting with a160eda212f92436016a07e9356e375791a43a95c53f1731fae3beeef020dee1 not found: ID does not exist"
Sep 18 21:06:15.818105 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.818087    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"]
Sep 18 21:06:15.822942 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:15.822922    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-76b5c79bc9-nftvh"]
Sep 18 21:06:17.460026 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:17.459978    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" path="/var/lib/kubelet/pods/bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7/volumes"
Sep 18 21:06:18.804150 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:18.804126    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4_c542399e-b39c-446c-8245-fc810ce8e44e/storage-initializer/0.log"
Sep 18 21:06:18.804440 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:18.804162    2566 generic.go:356] "Generic (PLEG): container finished" podID="c542399e-b39c-446c-8245-fc810ce8e44e" containerID="0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a" exitCode=1
Sep 18 21:06:18.804440 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:18.804240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4" event={"ID":"c542399e-b39c-446c-8245-fc810ce8e44e","Type":"ContainerDied","Data":"0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a"}
Sep 18 21:06:19.809718 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:19.809679    2566 generic.go:356] "Generic (PLEG): container finished" podID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerID="7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674" exitCode=0
Sep 18 21:06:19.810116 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:19.809750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" event={"ID":"a4f424dc-13ff-4300-84c4-39edb809d5ff","Type":"ContainerDied","Data":"7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674"}
Sep 18 21:06:19.811081 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:19.811065    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:06:19.811811 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:19.811796    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4_c542399e-b39c-446c-8245-fc810ce8e44e/storage-initializer/0.log"
Sep 18 21:06:19.811888 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:19.811844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4" event={"ID":"c542399e-b39c-446c-8245-fc810ce8e44e","Type":"ContainerStarted","Data":"bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb"}
Sep 18 21:06:21.764970 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:21.764936    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"]
Sep 18 21:06:21.765419 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:21.765217    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4" podUID="c542399e-b39c-446c-8245-fc810ce8e44e" containerName="storage-initializer" containerID="cri-o://bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb" gracePeriod=30
Sep 18 21:06:22.919667 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.919620    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"]
Sep 18 21:06:22.930225 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.930181    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kube-rbac-proxy"
Sep 18 21:06:22.930447 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.930422    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kube-rbac-proxy"
Sep 18 21:06:22.930579 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.930472    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="storage-initializer"
Sep 18 21:06:22.930579 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.930497    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="storage-initializer"
Sep 18 21:06:22.930579 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.930542    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container"
Sep 18 21:06:22.930579 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.930551    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container"
Sep 18 21:06:22.930798 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.930734    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kube-rbac-proxy"
Sep 18 21:06:22.930798 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.930749    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbcdff2c-e9e2-4bde-ae13-b6c5750c9fb7" containerName="kserve-container"
Sep 18 21:06:22.938691 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.938653    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"]
Sep 18 21:06:22.938890 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.938865    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:22.941773 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.941749    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 18 21:06:22.942579 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:22.942545    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 18 21:06:23.099041 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.098962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.099041 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.099037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zc7\" (UniqueName: \"kubernetes.io/projected/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-api-access-76zc7\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.099280 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.099085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.099280 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.099151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.099280 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.099193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.137992 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.137969    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4_c542399e-b39c-446c-8245-fc810ce8e44e/storage-initializer/1.log"
Sep 18 21:06:23.138492 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.138474    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4_c542399e-b39c-446c-8245-fc810ce8e44e/storage-initializer/0.log"
Sep 18 21:06:23.141802 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.141774    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:23.199980 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.199914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.199980 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.199949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zc7\" (UniqueName: \"kubernetes.io/projected/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-api-access-76zc7\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.200196 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.199988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.200196 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.200039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.200196 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.200080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.200576 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.200549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.200815 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.200785    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.200927 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.200815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.202834 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.202813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.209649 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.209620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zc7\" (UniqueName: \"kubernetes.io/projected/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-api-access-76zc7\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.251449 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.251417    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:23.300769 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.300732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj75w\" (UniqueName: \"kubernetes.io/projected/c542399e-b39c-446c-8245-fc810ce8e44e-kube-api-access-sj75w\") pod \"c542399e-b39c-446c-8245-fc810ce8e44e\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") "
Sep 18 21:06:23.300906 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.300784    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c542399e-b39c-446c-8245-fc810ce8e44e-kserve-provision-location\") pod \"c542399e-b39c-446c-8245-fc810ce8e44e\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") "
Sep 18 21:06:23.300906 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.300828    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c542399e-b39c-446c-8245-fc810ce8e44e-proxy-tls\") pod \"c542399e-b39c-446c-8245-fc810ce8e44e\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") "
Sep 18 21:06:23.300906 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.300889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c542399e-b39c-446c-8245-fc810ce8e44e-kube-rbac-proxy-sar-config\") pod \"c542399e-b39c-446c-8245-fc810ce8e44e\" (UID: \"c542399e-b39c-446c-8245-fc810ce8e44e\") "
Sep 18 21:06:23.301122 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.301084    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c542399e-b39c-446c-8245-fc810ce8e44e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c542399e-b39c-446c-8245-fc810ce8e44e" (UID: "c542399e-b39c-446c-8245-fc810ce8e44e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:06:23.301365 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.301338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c542399e-b39c-446c-8245-fc810ce8e44e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c542399e-b39c-446c-8245-fc810ce8e44e" (UID: "c542399e-b39c-446c-8245-fc810ce8e44e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:06:23.303189 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.303162    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c542399e-b39c-446c-8245-fc810ce8e44e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c542399e-b39c-446c-8245-fc810ce8e44e" (UID: "c542399e-b39c-446c-8245-fc810ce8e44e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:06:23.303299 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.303281    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c542399e-b39c-446c-8245-fc810ce8e44e-kube-api-access-sj75w" (OuterVolumeSpecName: "kube-api-access-sj75w") pod "c542399e-b39c-446c-8245-fc810ce8e44e" (UID: "c542399e-b39c-446c-8245-fc810ce8e44e"). InnerVolumeSpecName "kube-api-access-sj75w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:06:23.402102 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.402066    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj75w\" (UniqueName: \"kubernetes.io/projected/c542399e-b39c-446c-8245-fc810ce8e44e-kube-api-access-sj75w\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:06:23.402102 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.402099    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c542399e-b39c-446c-8245-fc810ce8e44e-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:06:23.402298 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.402112    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c542399e-b39c-446c-8245-fc810ce8e44e-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:06:23.402298 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.402126    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c542399e-b39c-446c-8245-fc810ce8e44e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:06:23.531804 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.531777    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"]
Sep 18 21:06:23.533613 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:06:23.533583    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca6cd20_ebce_4de1_9dbd_640ac1a27f80.slice/crio-dfbe0541f6a359066d4df675702ca8a3922dba60e7856c66e6c35bca4637ce4b WatchSource:0}: Error finding container dfbe0541f6a359066d4df675702ca8a3922dba60e7856c66e6c35bca4637ce4b: Status 404 returned error can't find the container with id dfbe0541f6a359066d4df675702ca8a3922dba60e7856c66e6c35bca4637ce4b
Sep 18 21:06:23.833426 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.833386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" event={"ID":"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80","Type":"ContainerStarted","Data":"dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba"}
Sep 18 21:06:23.833598 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.833460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" event={"ID":"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80","Type":"ContainerStarted","Data":"dfbe0541f6a359066d4df675702ca8a3922dba60e7856c66e6c35bca4637ce4b"}
Sep 18 21:06:23.835230 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.835207    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4_c542399e-b39c-446c-8245-fc810ce8e44e/storage-initializer/1.log"
Sep 18 21:06:23.835709 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.835688    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4_c542399e-b39c-446c-8245-fc810ce8e44e/storage-initializer/0.log"
Sep 18 21:06:23.835819 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.835730    2566 generic.go:356] "Generic (PLEG): container finished" podID="c542399e-b39c-446c-8245-fc810ce8e44e" containerID="bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb" exitCode=1
Sep 18 21:06:23.835819 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.835761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4" event={"ID":"c542399e-b39c-446c-8245-fc810ce8e44e","Type":"ContainerDied","Data":"bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb"}
Sep 18 21:06:23.835819 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.835802    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"
Sep 18 21:06:23.835980 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.835828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4" event={"ID":"c542399e-b39c-446c-8245-fc810ce8e44e","Type":"ContainerDied","Data":"9f8ed791a90ef90e45907d9b05cab638a4ba79f0df479702d5358d74230506ea"}
Sep 18 21:06:23.835980 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.835873    2566 scope.go:117] "RemoveContainer" containerID="bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb"
Sep 18 21:06:23.845897 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.845879    2566 scope.go:117] "RemoveContainer" containerID="0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a"
Sep 18 21:06:23.855527 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.855498    2566 scope.go:117] "RemoveContainer" containerID="bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb"
Sep 18 21:06:23.855804 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:06:23.855787    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb\": container with ID starting with bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb not found: ID does not exist" containerID="bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb"
Sep 18 21:06:23.855896 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.855813    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb"} err="failed to get container status \"bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb\": rpc error: code = NotFound desc = could not find container \"bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb\": container with ID starting with bdf4525e9ef8f385467ce8b17102fb7f80bd8e435ab94e0b3cd45b683377ffeb not found: ID does not exist"
Sep 18 21:06:23.855896 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.855831    2566 scope.go:117] "RemoveContainer" containerID="0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a"
Sep 18 21:06:23.856192 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:06:23.856149    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a\": container with ID starting with 0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a not found: ID does not exist" containerID="0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a"
Sep 18 21:06:23.856288 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.856196    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a"} err="failed to get container status \"0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a\": rpc error: code = NotFound desc = could not find container \"0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a\": container with ID starting with 0f49120a1b9ef8d382755102dbe5c464601588ec03fc1d2769cd4dcf181f911a not found: ID does not exist"
Sep 18 21:06:23.882323 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.882294    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"]
Sep 18 21:06:23.885615 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:23.885589    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-cqvw4"]
Sep 18 21:06:25.461602 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:25.461566    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c542399e-b39c-446c-8245-fc810ce8e44e" path="/var/lib/kubelet/pods/c542399e-b39c-446c-8245-fc810ce8e44e/volumes"
Sep 18 21:06:25.848806 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:25.848768    2566 generic.go:356] "Generic (PLEG): container finished" podID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerID="dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba" exitCode=0
Sep 18 21:06:25.848992 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:25.848845    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" event={"ID":"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80","Type":"ContainerDied","Data":"dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba"}
Sep 18 21:06:31.880660 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.880627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" event={"ID":"a4f424dc-13ff-4300-84c4-39edb809d5ff","Type":"ContainerStarted","Data":"2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1"}
Sep 18 21:06:31.881110 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.880692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" event={"ID":"a4f424dc-13ff-4300-84c4-39edb809d5ff","Type":"ContainerStarted","Data":"bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d"}
Sep 18 21:06:31.881110 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.880984    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:31.881110 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.881060    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:31.882513 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.882483    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 21:06:31.883132 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.883114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" event={"ID":"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80","Type":"ContainerStarted","Data":"6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39"}
Sep 18 21:06:31.883219 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.883160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" event={"ID":"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80","Type":"ContainerStarted","Data":"d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9"}
Sep 18 21:06:31.883441 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.883419    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:31.883525 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.883467    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:31.884517 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.884491    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 21:06:31.899296 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.899255    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podStartSLOduration=2.955372503 podStartE2EDuration="18.899243233s" totalImagesPullingTime="11.364680558s" totalInitContainerRuntime="4.579190172s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:31.8976271 +0000 UTC m=+2760.928553239" watchObservedRunningTime="2026-09-18 21:06:31.899243233 +0000 UTC m=+2760.930169373"
Sep 18 21:06:31.915025 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:31.914977    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podStartSLOduration=8.58747041 podStartE2EDuration="9.914966179s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.327495769s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:31.91333569 +0000 UTC m=+2760.944261830" watchObservedRunningTime="2026-09-18 21:06:31.914966179 +0000 UTC m=+2760.945892319"
Sep 18 21:06:32.888140 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:32.888087    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 21:06:32.888543 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:32.888265    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 21:06:37.892902 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:37.892874    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:06:37.893521 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:37.893492    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:06:37.893521 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:37.893502    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 21:06:37.893995 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:37.893974    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 21:06:47.893732 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:47.893695    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 21:06:47.894160 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:47.893972    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 21:06:57.893706 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:57.893610    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 21:06:57.894230 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:06:57.893876    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 21:07:07.893840 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:07.893793    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 21:07:07.894257 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:07.893932    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 21:07:17.893938 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:17.893901    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 21:07:17.894514 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:17.894497    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:07:25.278916 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:25.278876    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"]
Sep 18 21:07:25.279402 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:25.279252    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container" containerID="cri-o://bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d" gracePeriod=30
Sep 18 21:07:25.279402 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:25.279279    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kube-rbac-proxy" containerID="cri-o://2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1" gracePeriod=30
Sep 18 21:07:26.097392 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:26.097360    2566 generic.go:356] "Generic (PLEG): container finished" podID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerID="2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1" exitCode=2
Sep 18 21:07:26.097565 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:26.097407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" event={"ID":"a4f424dc-13ff-4300-84c4-39edb809d5ff","Type":"ContainerDied","Data":"2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1"}
Sep 18 21:07:27.888499 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:27.888452    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Sep 18 21:07:27.893762 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:27.893734    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 21:07:27.893913 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:27.893891    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 21:07:28.315122 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.315101    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:07:28.433251 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.433219    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzf88\" (UniqueName: \"kubernetes.io/projected/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-api-access-nzf88\") pod \"a4f424dc-13ff-4300-84c4-39edb809d5ff\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") "
Sep 18 21:07:28.433419 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.433268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-rbac-proxy-sar-config\") pod \"a4f424dc-13ff-4300-84c4-39edb809d5ff\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") "
Sep 18 21:07:28.433419 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.433295    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-cabundle-cert\") pod \"a4f424dc-13ff-4300-84c4-39edb809d5ff\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") "
Sep 18 21:07:28.433419 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.433358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f424dc-13ff-4300-84c4-39edb809d5ff-kserve-provision-location\") pod \"a4f424dc-13ff-4300-84c4-39edb809d5ff\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") "
Sep 18 21:07:28.433419 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.433380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f424dc-13ff-4300-84c4-39edb809d5ff-proxy-tls\") pod \"a4f424dc-13ff-4300-84c4-39edb809d5ff\" (UID: \"a4f424dc-13ff-4300-84c4-39edb809d5ff\") "
Sep 18 21:07:28.433717 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.433689    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4f424dc-13ff-4300-84c4-39edb809d5ff" (UID: "a4f424dc-13ff-4300-84c4-39edb809d5ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:07:28.433785 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.433708    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4f424dc-13ff-4300-84c4-39edb809d5ff" (UID: "a4f424dc-13ff-4300-84c4-39edb809d5ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:07:28.435377 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.435351    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f424dc-13ff-4300-84c4-39edb809d5ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4f424dc-13ff-4300-84c4-39edb809d5ff" (UID: "a4f424dc-13ff-4300-84c4-39edb809d5ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:07:28.435480 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.435430    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-api-access-nzf88" (OuterVolumeSpecName: "kube-api-access-nzf88") pod "a4f424dc-13ff-4300-84c4-39edb809d5ff" (UID: "a4f424dc-13ff-4300-84c4-39edb809d5ff"). InnerVolumeSpecName "kube-api-access-nzf88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:07:28.447323 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.447266    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f424dc-13ff-4300-84c4-39edb809d5ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4f424dc-13ff-4300-84c4-39edb809d5ff" (UID: "a4f424dc-13ff-4300-84c4-39edb809d5ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:07:28.534733 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.534707    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f424dc-13ff-4300-84c4-39edb809d5ff-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:28.534733 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.534729    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f424dc-13ff-4300-84c4-39edb809d5ff-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:28.534733 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.534740    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzf88\" (UniqueName: \"kubernetes.io/projected/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-api-access-nzf88\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:28.534930 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.534748    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:28.534930 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:28.534757    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f424dc-13ff-4300-84c4-39edb809d5ff-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:29.110581 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.110550    2566 generic.go:356] "Generic (PLEG): container finished" podID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerID="bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d" exitCode=0
Sep 18 21:07:29.111071 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.110610    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"
Sep 18 21:07:29.111071 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.110615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" event={"ID":"a4f424dc-13ff-4300-84c4-39edb809d5ff","Type":"ContainerDied","Data":"bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d"}
Sep 18 21:07:29.111071 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.110734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw" event={"ID":"a4f424dc-13ff-4300-84c4-39edb809d5ff","Type":"ContainerDied","Data":"930bb55d311deef82f2d9783f42d8cc85e950affbd0109270d0131d16da7a1ee"}
Sep 18 21:07:29.111071 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.110760    2566 scope.go:117] "RemoveContainer" containerID="2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1"
Sep 18 21:07:29.119413 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.119393    2566 scope.go:117] "RemoveContainer" containerID="bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d"
Sep 18 21:07:29.126836 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.126819    2566 scope.go:117] "RemoveContainer" containerID="7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674"
Sep 18 21:07:29.133703 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.133681    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"]
Sep 18 21:07:29.134138 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.134125    2566 scope.go:117] "RemoveContainer" containerID="2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1"
Sep 18 21:07:29.134397 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:07:29.134380    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1\": container with ID starting with 2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1 not found: ID does not exist" containerID="2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1"
Sep 18 21:07:29.134480 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.134409    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1"} err="failed to get container status \"2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1\": rpc error: code = NotFound desc = could not find container \"2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1\": container with ID starting with 2ec86aefcb2f04ae744af88fef144b27dff8a2da5587992225dc08b2ad6625a1 not found: ID does not exist"
Sep 18 21:07:29.134480 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.134431    2566 scope.go:117] "RemoveContainer" containerID="bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d"
Sep 18 21:07:29.134654 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:07:29.134638    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d\": container with ID starting with bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d not found: ID does not exist" containerID="bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d"
Sep 18 21:07:29.134721 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.134661    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d"} err="failed to get container status \"bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d\": rpc error: code = NotFound desc = could not find container \"bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d\": container with ID starting with bff6d8d43ec4923850e8063c84cccebc302e2a6f9e33443898c6edd8f1a2725d not found: ID does not exist"
Sep 18 21:07:29.134721 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.134681    2566 scope.go:117] "RemoveContainer" containerID="7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674"
Sep 18 21:07:29.134877 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:07:29.134862    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674\": container with ID starting with 7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674 not found: ID does not exist" containerID="7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674"
Sep 18 21:07:29.134934 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.134884    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674"} err="failed to get container status \"7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674\": rpc error: code = NotFound desc = could not find container \"7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674\": container with ID starting with 7420edf00176f95a15e21b08c921318aca10b20a1e6db1bda8f3525145912674 not found: ID does not exist"
Sep 18 21:07:29.137782 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.137762    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lj5kw"]
Sep 18 21:07:29.460333 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:29.460258    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" path="/var/lib/kubelet/pods/a4f424dc-13ff-4300-84c4-39edb809d5ff/volumes"
Sep 18 21:07:30.490636 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.490599    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"]
Sep 18 21:07:30.491064 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491048    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container"
Sep 18 21:07:30.491121 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491066    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container"
Sep 18 21:07:30.491121 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491078    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c542399e-b39c-446c-8245-fc810ce8e44e" containerName="storage-initializer"
Sep 18 21:07:30.491121 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491087    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c542399e-b39c-446c-8245-fc810ce8e44e" containerName="storage-initializer"
Sep 18 21:07:30.491121 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491096    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="storage-initializer"
Sep 18 21:07:30.491121 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491105    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="storage-initializer"
Sep 18 21:07:30.491121 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491115    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kube-rbac-proxy"
Sep 18 21:07:30.491306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491124    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kube-rbac-proxy"
Sep 18 21:07:30.491306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491134    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c542399e-b39c-446c-8245-fc810ce8e44e" containerName="storage-initializer"
Sep 18 21:07:30.491306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491142    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c542399e-b39c-446c-8245-fc810ce8e44e" containerName="storage-initializer"
Sep 18 21:07:30.491306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491234    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c542399e-b39c-446c-8245-fc810ce8e44e" containerName="storage-initializer"
Sep 18 21:07:30.491306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491244    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c542399e-b39c-446c-8245-fc810ce8e44e" containerName="storage-initializer"
Sep 18 21:07:30.491306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491253    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kserve-container"
Sep 18 21:07:30.491306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.491263    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f424dc-13ff-4300-84c4-39edb809d5ff" containerName="kube-rbac-proxy"
Sep 18 21:07:30.498884 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.498861    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.502005 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.501978    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 21:07:30.502005 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.501987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 18 21:07:30.502294 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.502274    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 18 21:07:30.506953 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.506927    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"]
Sep 18 21:07:30.652854 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.652820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.653058 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.652869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.653058 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.652931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf8635e0-b723-4627-9f8d-e23a31c36808-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.653058 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.652979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf8635e0-b723-4627-9f8d-e23a31c36808-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.653178 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.653075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpg46\" (UniqueName: \"kubernetes.io/projected/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-api-access-kpg46\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.753753 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.753674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpg46\" (UniqueName: \"kubernetes.io/projected/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-api-access-kpg46\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.753753 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.753716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.753922 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.753785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.753922 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.753821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf8635e0-b723-4627-9f8d-e23a31c36808-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.753922 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.753847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf8635e0-b723-4627-9f8d-e23a31c36808-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.754302 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.754280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf8635e0-b723-4627-9f8d-e23a31c36808-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.754462 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.754438    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.754462 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.754452    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.756332 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.756306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf8635e0-b723-4627-9f8d-e23a31c36808-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.762424 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.762402    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpg46\" (UniqueName: \"kubernetes.io/projected/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-api-access-kpg46\") pod \"isvc-paddle-runtime-predictor-58b99fff76-txdxc\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:30.810373 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:30.810348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:31.085043 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:31.085002    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"]
Sep 18 21:07:31.087008 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:07:31.086980    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8635e0_b723_4627_9f8d_e23a31c36808.slice/crio-9078ba58e02b6aa3f16578060765c147b44b8e1955760d9398501b2ea1ee41e5 WatchSource:0}: Error finding container 9078ba58e02b6aa3f16578060765c147b44b8e1955760d9398501b2ea1ee41e5: Status 404 returned error can't find the container with id 9078ba58e02b6aa3f16578060765c147b44b8e1955760d9398501b2ea1ee41e5
Sep 18 21:07:31.121168 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:31.121141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" event={"ID":"cf8635e0-b723-4627-9f8d-e23a31c36808","Type":"ContainerStarted","Data":"9078ba58e02b6aa3f16578060765c147b44b8e1955760d9398501b2ea1ee41e5"}
Sep 18 21:07:32.127443 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:32.127356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" event={"ID":"cf8635e0-b723-4627-9f8d-e23a31c36808","Type":"ContainerStarted","Data":"70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7"}
Sep 18 21:07:36.144079 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:36.143979    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerID="70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7" exitCode=0
Sep 18 21:07:36.144079 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:36.144048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" event={"ID":"cf8635e0-b723-4627-9f8d-e23a31c36808","Type":"ContainerDied","Data":"70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7"}
Sep 18 21:07:37.151144 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:37.151112    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" event={"ID":"cf8635e0-b723-4627-9f8d-e23a31c36808","Type":"ContainerStarted","Data":"e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f"}
Sep 18 21:07:37.151526 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:37.151173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" event={"ID":"cf8635e0-b723-4627-9f8d-e23a31c36808","Type":"ContainerStarted","Data":"248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e"}
Sep 18 21:07:37.151526 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:37.151353    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:37.171617 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:37.171569    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podStartSLOduration=2.46974955 podStartE2EDuration="7.171552687s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.701803137s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:37.16954564 +0000 UTC m=+2826.200471781" watchObservedRunningTime="2026-09-18 21:07:37.171552687 +0000 UTC m=+2826.202478827"
Sep 18 21:07:37.893564 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:37.893530    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 21:07:38.155573 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:38.155492    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:38.156930 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:38.156902    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 18 21:07:39.160576 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:39.160538    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 18 21:07:44.165443 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:44.165413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:07:44.165895 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:44.165867    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 18 21:07:47.894879 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:47.894850    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:07:52.927441 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:52.927398    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"]
Sep 18 21:07:52.927915 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:52.927752    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container" containerID="cri-o://d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9" gracePeriod=30
Sep 18 21:07:52.927915 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:52.927793    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kube-rbac-proxy" containerID="cri-o://6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39" gracePeriod=30
Sep 18 21:07:53.219450 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:53.219366    2566 generic.go:356] "Generic (PLEG): container finished" podID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerID="6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39" exitCode=2
Sep 18 21:07:53.219450 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:53.219432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" event={"ID":"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80","Type":"ContainerDied","Data":"6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39"}
Sep 18 21:07:54.165854 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:54.165815    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 18 21:07:57.666414 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.666390    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:07:57.770554 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.770533    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76zc7\" (UniqueName: \"kubernetes.io/projected/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-api-access-76zc7\") pod \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") "
Sep 18 21:07:57.770730 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.770610    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-cabundle-cert\") pod \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") "
Sep 18 21:07:57.770730 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.770646    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-proxy-tls\") pod \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") "
Sep 18 21:07:57.770730 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.770671    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-rbac-proxy-sar-config\") pod \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") "
Sep 18 21:07:57.770730 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.770702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kserve-provision-location\") pod \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\" (UID: \"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80\") "
Sep 18 21:07:57.771137 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.771102    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" (UID: "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:07:57.771268 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.771157    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" (UID: "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:07:57.771268 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.771165    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" (UID: "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:07:57.772684 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.772657    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" (UID: "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:07:57.772786 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.772710    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-api-access-76zc7" (OuterVolumeSpecName: "kube-api-access-76zc7") pod "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" (UID: "2ca6cd20-ebce-4de1-9dbd-640ac1a27f80"). InnerVolumeSpecName "kube-api-access-76zc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:07:57.871501 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.871475    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:57.871501 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.871497    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:57.871501 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.871508    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76zc7\" (UniqueName: \"kubernetes.io/projected/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-kube-api-access-76zc7\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:57.871692 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.871518    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:57.871692 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:57.871527    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:07:58.241799 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.241711    2566 generic.go:356] "Generic (PLEG): container finished" podID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerID="d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9" exitCode=0
Sep 18 21:07:58.241799 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.241766    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"
Sep 18 21:07:58.241799 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.241774    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" event={"ID":"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80","Type":"ContainerDied","Data":"d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9"}
Sep 18 21:07:58.242044 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.241819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp" event={"ID":"2ca6cd20-ebce-4de1-9dbd-640ac1a27f80","Type":"ContainerDied","Data":"dfbe0541f6a359066d4df675702ca8a3922dba60e7856c66e6c35bca4637ce4b"}
Sep 18 21:07:58.242044 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.241846    2566 scope.go:117] "RemoveContainer" containerID="6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39"
Sep 18 21:07:58.250229 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.250207    2566 scope.go:117] "RemoveContainer" containerID="d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9"
Sep 18 21:07:58.257150 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.257134    2566 scope.go:117] "RemoveContainer" containerID="dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba"
Sep 18 21:07:58.263618 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.263597    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"]
Sep 18 21:07:58.264387 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.264375    2566 scope.go:117] "RemoveContainer" containerID="6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39"
Sep 18 21:07:58.264624 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:07:58.264605    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39\": container with ID starting with 6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39 not found: ID does not exist" containerID="6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39"
Sep 18 21:07:58.264683 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.264630    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39"} err="failed to get container status \"6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39\": rpc error: code = NotFound desc = could not find container \"6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39\": container with ID starting with 6e690ca42654c44431a50b5345b90c7a69cb368a5021e7f80906502fa4c92f39 not found: ID does not exist"
Sep 18 21:07:58.264683 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.264646    2566 scope.go:117] "RemoveContainer" containerID="d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9"
Sep 18 21:07:58.264892 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:07:58.264873    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9\": container with ID starting with d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9 not found: ID does not exist" containerID="d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9"
Sep 18 21:07:58.264934 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.264897    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9"} err="failed to get container status \"d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9\": rpc error: code = NotFound desc = could not find container \"d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9\": container with ID starting with d0b65898c49cda80b9ef1b45cd9809b7233744b873b5e4ec71e84fa1d6c6aba9 not found: ID does not exist"
Sep 18 21:07:58.264934 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.264914    2566 scope.go:117] "RemoveContainer" containerID="dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba"
Sep 18 21:07:58.265303 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:07:58.265228    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba\": container with ID starting with dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba not found: ID does not exist" containerID="dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba"
Sep 18 21:07:58.265303 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.265259    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba"} err="failed to get container status \"dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba\": rpc error: code = NotFound desc = could not find container \"dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba\": container with ID starting with dda36fe17fbb43139313759d6b36fceeacf80be8c426a2917a9f428a16030cba not found: ID does not exist"
Sep 18 21:07:58.268615 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:58.268587    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-9xlsp"]
Sep 18 21:07:59.459997 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:07:59.459960    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" path="/var/lib/kubelet/pods/2ca6cd20-ebce-4de1-9dbd-640ac1a27f80/volumes"
Sep 18 21:08:04.166214 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:04.166177    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 18 21:08:05.157086 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.157050    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"]
Sep 18 21:08:05.157490 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.157476    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="storage-initializer"
Sep 18 21:08:05.157540 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.157493    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="storage-initializer"
Sep 18 21:08:05.157540 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.157509    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container"
Sep 18 21:08:05.157540 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.157518    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container"
Sep 18 21:08:05.157643 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.157549    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kube-rbac-proxy"
Sep 18 21:08:05.157643 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.157558    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kube-rbac-proxy"
Sep 18 21:08:05.157643 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.157619    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kserve-container"
Sep 18 21:08:05.157643 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.157631    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca6cd20-ebce-4de1-9dbd-640ac1a27f80" containerName="kube-rbac-proxy"
Sep 18 21:08:05.162579 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.162554    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.165267 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.165241    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 18 21:08:05.165376 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.165277    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 21:08:05.165376 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.165241    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 18 21:08:05.171229 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.171207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"]
Sep 18 21:08:05.227920 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.227889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc74968-f6bb-4ff4-b79b-26b8982b0247-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.228079 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.227927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.228079 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.227951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.228079 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.228035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.228206 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.228095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv52w\" (UniqueName: \"kubernetes.io/projected/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-api-access-fv52w\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.328649 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.328616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc74968-f6bb-4ff4-b79b-26b8982b0247-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.328649 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.328652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.328866 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.328670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.328866 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.328695    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.328866 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.328741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv52w\" (UniqueName: \"kubernetes.io/projected/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-api-access-fv52w\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.329235 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.329210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.329412 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.329393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.329481 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.329417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.331096 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.331080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc74968-f6bb-4ff4-b79b-26b8982b0247-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.336880 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.336862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv52w\" (UniqueName: \"kubernetes.io/projected/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-api-access-fv52w\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.475427 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.475351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:05.744650 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:05.744621    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"]
Sep 18 21:08:05.746949 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:08:05.746923    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc74968_f6bb_4ff4_b79b_26b8982b0247.slice/crio-2f2a21631c35b3b6f712c7bca444da9a2bcab135243e5ef7a26b64d5c5802bab WatchSource:0}: Error finding container 2f2a21631c35b3b6f712c7bca444da9a2bcab135243e5ef7a26b64d5c5802bab: Status 404 returned error can't find the container with id 2f2a21631c35b3b6f712c7bca444da9a2bcab135243e5ef7a26b64d5c5802bab
Sep 18 21:08:06.275003 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:06.274970    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" event={"ID":"6dc74968-f6bb-4ff4-b79b-26b8982b0247","Type":"ContainerStarted","Data":"fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df"}
Sep 18 21:08:06.275457 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:06.275058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" event={"ID":"6dc74968-f6bb-4ff4-b79b-26b8982b0247","Type":"ContainerStarted","Data":"2f2a21631c35b3b6f712c7bca444da9a2bcab135243e5ef7a26b64d5c5802bab"}
Sep 18 21:08:07.280687 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:07.280653    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerID="fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df" exitCode=0
Sep 18 21:08:07.281179 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:07.280704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" event={"ID":"6dc74968-f6bb-4ff4-b79b-26b8982b0247","Type":"ContainerDied","Data":"fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df"}
Sep 18 21:08:08.286745 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:08.286702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" event={"ID":"6dc74968-f6bb-4ff4-b79b-26b8982b0247","Type":"ContainerStarted","Data":"4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a"}
Sep 18 21:08:08.287167 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:08.286784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" event={"ID":"6dc74968-f6bb-4ff4-b79b-26b8982b0247","Type":"ContainerStarted","Data":"b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447"}
Sep 18 21:08:08.287167 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:08.286825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:08.305163 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:08.305121    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podStartSLOduration=2.2838562749999998 podStartE2EDuration="3.30510787s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.021251595s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:08.30375798 +0000 UTC m=+2857.334684131" watchObservedRunningTime="2026-09-18 21:08:08.30510787 +0000 UTC m=+2857.336034065"
Sep 18 21:08:09.290924 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:09.290890    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:09.292008 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:09.291976    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:08:10.295277 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:10.295239    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:08:14.166911 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:14.166862    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 18 21:08:15.299978 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:15.299951    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:08:15.300428 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:15.300405    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:08:24.166822 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:24.166748    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:08:25.300700 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:25.300660    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:08:31.885037 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:31.884986    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"]
Sep 18 21:08:31.885431 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:31.885258    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container" containerID="cri-o://248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e" gracePeriod=30
Sep 18 21:08:31.885431 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:31.885298    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kube-rbac-proxy" containerID="cri-o://e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f" gracePeriod=30
Sep 18 21:08:32.386907 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:32.386872    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerID="e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f" exitCode=2
Sep 18 21:08:32.387090 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:32.386914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" event={"ID":"cf8635e0-b723-4627-9f8d-e23a31c36808","Type":"ContainerDied","Data":"e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f"}
Sep 18 21:08:34.161755 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.161715    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Sep 18 21:08:34.166357 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.166314    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 18 21:08:34.829341 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.829319    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:08:34.869093 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.869066    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf8635e0-b723-4627-9f8d-e23a31c36808-proxy-tls\") pod \"cf8635e0-b723-4627-9f8d-e23a31c36808\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") "
Sep 18 21:08:34.869241 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.869133    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpg46\" (UniqueName: \"kubernetes.io/projected/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-api-access-kpg46\") pod \"cf8635e0-b723-4627-9f8d-e23a31c36808\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") "
Sep 18 21:08:34.869241 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.869176    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf8635e0-b723-4627-9f8d-e23a31c36808-kserve-provision-location\") pod \"cf8635e0-b723-4627-9f8d-e23a31c36808\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") "
Sep 18 21:08:34.869241 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.869210    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-cabundle-cert\") pod \"cf8635e0-b723-4627-9f8d-e23a31c36808\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") "
Sep 18 21:08:34.869409 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.869251    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-rbac-proxy-sar-config\") pod \"cf8635e0-b723-4627-9f8d-e23a31c36808\" (UID: \"cf8635e0-b723-4627-9f8d-e23a31c36808\") "
Sep 18 21:08:34.869644 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.869611    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf8635e0-b723-4627-9f8d-e23a31c36808" (UID: "cf8635e0-b723-4627-9f8d-e23a31c36808"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:08:34.869770 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.869682    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf8635e0-b723-4627-9f8d-e23a31c36808" (UID: "cf8635e0-b723-4627-9f8d-e23a31c36808"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:08:34.871646 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.871622    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf8635e0-b723-4627-9f8d-e23a31c36808-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf8635e0-b723-4627-9f8d-e23a31c36808" (UID: "cf8635e0-b723-4627-9f8d-e23a31c36808"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:08:34.871740 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.871647    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-api-access-kpg46" (OuterVolumeSpecName: "kube-api-access-kpg46") pod "cf8635e0-b723-4627-9f8d-e23a31c36808" (UID: "cf8635e0-b723-4627-9f8d-e23a31c36808"). InnerVolumeSpecName "kube-api-access-kpg46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:08:34.878786 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.878763    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8635e0-b723-4627-9f8d-e23a31c36808-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf8635e0-b723-4627-9f8d-e23a31c36808" (UID: "cf8635e0-b723-4627-9f8d-e23a31c36808"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:08:34.970202 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.970150    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:08:34.970202 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.970169    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf8635e0-b723-4627-9f8d-e23a31c36808-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:08:34.970202 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.970180    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpg46\" (UniqueName: \"kubernetes.io/projected/cf8635e0-b723-4627-9f8d-e23a31c36808-kube-api-access-kpg46\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:08:34.970202 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.970188    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf8635e0-b723-4627-9f8d-e23a31c36808-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:08:34.970202 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:34.970198    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf8635e0-b723-4627-9f8d-e23a31c36808-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:08:35.300444 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.300411    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:08:35.400743 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.400710    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerID="248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e" exitCode=0
Sep 18 21:08:35.400899 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.400769    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"
Sep 18 21:08:35.400899 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.400796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" event={"ID":"cf8635e0-b723-4627-9f8d-e23a31c36808","Type":"ContainerDied","Data":"248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e"}
Sep 18 21:08:35.400899 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.400881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc" event={"ID":"cf8635e0-b723-4627-9f8d-e23a31c36808","Type":"ContainerDied","Data":"9078ba58e02b6aa3f16578060765c147b44b8e1955760d9398501b2ea1ee41e5"}
Sep 18 21:08:35.401033 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.400910    2566 scope.go:117] "RemoveContainer" containerID="e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f"
Sep 18 21:08:35.409794 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.409772    2566 scope.go:117] "RemoveContainer" containerID="248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e"
Sep 18 21:08:35.416847 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.416829    2566 scope.go:117] "RemoveContainer" containerID="70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7"
Sep 18 21:08:35.422252 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.422228    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"]
Sep 18 21:08:35.424921 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.424731    2566 scope.go:117] "RemoveContainer" containerID="e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f"
Sep 18 21:08:35.425070 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:08:35.425050    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f\": container with ID starting with e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f not found: ID does not exist" containerID="e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f"
Sep 18 21:08:35.425128 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.425078    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f"} err="failed to get container status \"e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f\": rpc error: code = NotFound desc = could not find container \"e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f\": container with ID starting with e058880f53d3088cf01384595f0331e7516085a0e90a9944f1df12ba5ed8048f not found: ID does not exist"
Sep 18 21:08:35.425128 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.425095    2566 scope.go:117] "RemoveContainer" containerID="248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e"
Sep 18 21:08:35.425418 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:08:35.425400    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e\": container with ID starting with 248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e not found: ID does not exist" containerID="248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e"
Sep 18 21:08:35.425488 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.425428    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e"} err="failed to get container status \"248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e\": rpc error: code = NotFound desc = could not find container \"248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e\": container with ID starting with 248ef7420d4621c52ff8836ce035ff77ac69d90f377930299c27858490e72f5e not found: ID does not exist"
Sep 18 21:08:35.425488 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.425451    2566 scope.go:117] "RemoveContainer" containerID="70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7"
Sep 18 21:08:35.425710 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:08:35.425694    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7\": container with ID starting with 70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7 not found: ID does not exist" containerID="70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7"
Sep 18 21:08:35.425753 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.425715    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7"} err="failed to get container status \"70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7\": rpc error: code = NotFound desc = could not find container \"70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7\": container with ID starting with 70305997c3d5a8f561cd5d7eabecb587291d8ee4a800a3f23cc12a1b15b347a7 not found: ID does not exist"
Sep 18 21:08:35.426053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.426031    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-txdxc"]
Sep 18 21:08:35.459335 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:35.459313    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" path="/var/lib/kubelet/pods/cf8635e0-b723-4627-9f8d-e23a31c36808/volumes"
Sep 18 21:08:37.092690 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.092661    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"]
Sep 18 21:08:37.093053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.092942    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container"
Sep 18 21:08:37.093053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.092952    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container"
Sep 18 21:08:37.093053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.092972    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="storage-initializer"
Sep 18 21:08:37.093053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.092980    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="storage-initializer"
Sep 18 21:08:37.093053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.092989    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kube-rbac-proxy"
Sep 18 21:08:37.093053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.092998    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kube-rbac-proxy"
Sep 18 21:08:37.093251 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.093072    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kube-rbac-proxy"
Sep 18 21:08:37.093251 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.093079    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf8635e0-b723-4627-9f8d-e23a31c36808" containerName="kserve-container"
Sep 18 21:08:37.097742 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.097722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.100325 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.100301    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 18 21:08:37.100440 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.100359    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 21:08:37.100591 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.100569    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 21:08:37.107075 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.107053    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"]
Sep 18 21:08:37.186783 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.186748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.186783 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.186784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.186964 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.186809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.186964 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.186829    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzw4\" (UniqueName: \"kubernetes.io/projected/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-api-access-fqzw4\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.186964 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.186857    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.287742 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.287714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.287888 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.287747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.287888 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.287776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.288002 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.287893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzw4\" (UniqueName: \"kubernetes.io/projected/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-api-access-fqzw4\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.288002 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.287948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.288420 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.288392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.288503 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.288416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.288503 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.288443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.290182 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.290166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.296334 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.296315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzw4\" (UniqueName: \"kubernetes.io/projected/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-api-access-fqzw4\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.408787 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.408707    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:37.679050 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:37.678952    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"]
Sep 18 21:08:37.682542 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:08:37.682512    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d105f6_86ec_4572_a6ec_ab46e9d2f0b6.slice/crio-2f53b6419126374c94fa9754e0ff16453bfb51935304030455a37d6ae95422f1 WatchSource:0}: Error finding container 2f53b6419126374c94fa9754e0ff16453bfb51935304030455a37d6ae95422f1: Status 404 returned error can't find the container with id 2f53b6419126374c94fa9754e0ff16453bfb51935304030455a37d6ae95422f1
Sep 18 21:08:38.418484 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:38.418448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" event={"ID":"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6","Type":"ContainerStarted","Data":"096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778"}
Sep 18 21:08:38.418871 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:38.418511    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" event={"ID":"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6","Type":"ContainerStarted","Data":"2f53b6419126374c94fa9754e0ff16453bfb51935304030455a37d6ae95422f1"}
Sep 18 21:08:42.442893 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:42.442860    2566 generic.go:356] "Generic (PLEG): container finished" podID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerID="096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778" exitCode=0
Sep 18 21:08:42.443301 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:42.442918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" event={"ID":"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6","Type":"ContainerDied","Data":"096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778"}
Sep 18 21:08:43.449136 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:43.449103    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" event={"ID":"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6","Type":"ContainerStarted","Data":"281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090"}
Sep 18 21:08:43.449537 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:43.449169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" event={"ID":"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6","Type":"ContainerStarted","Data":"d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a"}
Sep 18 21:08:43.449537 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:43.449375    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:43.471570 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:43.471508    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podStartSLOduration=2.235058543 podStartE2EDuration="6.471488032s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.236429489s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:43.46796179 +0000 UTC m=+2892.498887942" watchObservedRunningTime="2026-09-18 21:08:43.471488032 +0000 UTC m=+2892.502414174"
Sep 18 21:08:44.452782 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:44.452746    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:44.454147 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:44.454124    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 18 21:08:45.301230 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:45.301195    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:08:45.456726 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:45.456687    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 18 21:08:50.461108 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:50.461078    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:08:50.461591 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:50.461564    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 18 21:08:55.300868 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:08:55.300829    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:09:00.461622 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:00.461578    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 18 21:09:05.300915 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:05.300877    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:09:10.461531 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:10.461496    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 18 21:09:15.300520 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:15.300484    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:09:20.462497 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:20.462454    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 18 21:09:25.301686 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:25.301657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:09:30.462155 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:30.462126    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:09:35.180092 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:35.179985    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"]
Sep 18 21:09:35.180500 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:35.180239    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" containerID="cri-o://b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447" gracePeriod=30
Sep 18 21:09:35.180500 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:35.180289    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kube-rbac-proxy" containerID="cri-o://4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a" gracePeriod=30
Sep 18 21:09:35.296171 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:35.296142    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Sep 18 21:09:35.300752 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:35.300715    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Sep 18 21:09:35.656976 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:35.656946    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerID="4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a" exitCode=2
Sep 18 21:09:35.657161 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:35.657035    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" event={"ID":"6dc74968-f6bb-4ff4-b79b-26b8982b0247","Type":"ContainerDied","Data":"4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a"}
Sep 18 21:09:38.890034 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:38.889985    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"]
Sep 18 21:09:38.890418 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:38.890291    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container" containerID="cri-o://d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a" gracePeriod=30
Sep 18 21:09:38.890418 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:38.890320    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kube-rbac-proxy" containerID="cri-o://281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090" gracePeriod=30
Sep 18 21:09:39.678115 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:39.678077    2566 generic.go:356] "Generic (PLEG): container finished" podID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerID="281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090" exitCode=2
Sep 18 21:09:39.678284 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:39.678144    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" event={"ID":"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6","Type":"ContainerDied","Data":"281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090"}
Sep 18 21:09:40.110127 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.110093    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:09:40.190386 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-rbac-proxy-sar-config\") pod \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") "
Sep 18 21:09:40.190566 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190413    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kserve-provision-location\") pod \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") "
Sep 18 21:09:40.190566 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190436    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv52w\" (UniqueName: \"kubernetes.io/projected/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-api-access-fv52w\") pod \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") "
Sep 18 21:09:40.190566 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190550    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-cabundle-cert\") pod \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") "
Sep 18 21:09:40.190742 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190607    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc74968-f6bb-4ff4-b79b-26b8982b0247-proxy-tls\") pod \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\" (UID: \"6dc74968-f6bb-4ff4-b79b-26b8982b0247\") "
Sep 18 21:09:40.190742 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190709    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6dc74968-f6bb-4ff4-b79b-26b8982b0247" (UID: "6dc74968-f6bb-4ff4-b79b-26b8982b0247"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:09:40.190830 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190745    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6dc74968-f6bb-4ff4-b79b-26b8982b0247" (UID: "6dc74968-f6bb-4ff4-b79b-26b8982b0247"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:09:40.190891 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6dc74968-f6bb-4ff4-b79b-26b8982b0247" (UID: "6dc74968-f6bb-4ff4-b79b-26b8982b0247"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:09:40.190891 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190884    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:40.190977 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.190903    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:40.192640 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.192621    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc74968-f6bb-4ff4-b79b-26b8982b0247-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6dc74968-f6bb-4ff4-b79b-26b8982b0247" (UID: "6dc74968-f6bb-4ff4-b79b-26b8982b0247"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:09:40.192640 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.192627    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-api-access-fv52w" (OuterVolumeSpecName: "kube-api-access-fv52w") pod "6dc74968-f6bb-4ff4-b79b-26b8982b0247" (UID: "6dc74968-f6bb-4ff4-b79b-26b8982b0247"). InnerVolumeSpecName "kube-api-access-fv52w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:09:40.291612 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.291585    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv52w\" (UniqueName: \"kubernetes.io/projected/6dc74968-f6bb-4ff4-b79b-26b8982b0247-kube-api-access-fv52w\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:40.291612 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.291609    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc74968-f6bb-4ff4-b79b-26b8982b0247-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:40.291612 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.291618    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc74968-f6bb-4ff4-b79b-26b8982b0247-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:40.457565 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.457530    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.63:8643/healthz\": dial tcp 10.133.0.63:8643: connect: connection refused"
Sep 18 21:09:40.461882 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.461860    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 18 21:09:40.684974 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.684884    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerID="b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447" exitCode=0
Sep 18 21:09:40.684974 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.684950    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"
Sep 18 21:09:40.685179 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.684975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" event={"ID":"6dc74968-f6bb-4ff4-b79b-26b8982b0247","Type":"ContainerDied","Data":"b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447"}
Sep 18 21:09:40.685179 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.685054    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx" event={"ID":"6dc74968-f6bb-4ff4-b79b-26b8982b0247","Type":"ContainerDied","Data":"2f2a21631c35b3b6f712c7bca444da9a2bcab135243e5ef7a26b64d5c5802bab"}
Sep 18 21:09:40.685179 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.685088    2566 scope.go:117] "RemoveContainer" containerID="4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a"
Sep 18 21:09:40.693905 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.693885    2566 scope.go:117] "RemoveContainer" containerID="b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447"
Sep 18 21:09:40.701419 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.701396    2566 scope.go:117] "RemoveContainer" containerID="fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df"
Sep 18 21:09:40.707095 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.707072    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"]
Sep 18 21:09:40.708675 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.708657    2566 scope.go:117] "RemoveContainer" containerID="4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a"
Sep 18 21:09:40.708932 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:09:40.708916    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a\": container with ID starting with 4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a not found: ID does not exist" containerID="4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a"
Sep 18 21:09:40.708997 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.708939    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a"} err="failed to get container status \"4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a\": rpc error: code = NotFound desc = could not find container \"4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a\": container with ID starting with 4698a07571f83513b19e49d36938f99ec1561f1f0da5e40bb43890a77c4cb56a not found: ID does not exist"
Sep 18 21:09:40.708997 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.708954    2566 scope.go:117] "RemoveContainer" containerID="b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447"
Sep 18 21:09:40.709226 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:09:40.709211    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447\": container with ID starting with b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447 not found: ID does not exist" containerID="b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447"
Sep 18 21:09:40.709271 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.709232    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447"} err="failed to get container status \"b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447\": rpc error: code = NotFound desc = could not find container \"b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447\": container with ID starting with b0637fd6916e3a99892035fd05d708fb990a2942de6e0d2b44816c286c765447 not found: ID does not exist"
Sep 18 21:09:40.709271 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.709246    2566 scope.go:117] "RemoveContainer" containerID="fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df"
Sep 18 21:09:40.709499 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:09:40.709481    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df\": container with ID starting with fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df not found: ID does not exist" containerID="fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df"
Sep 18 21:09:40.709548 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.709503    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df"} err="failed to get container status \"fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df\": rpc error: code = NotFound desc = could not find container \"fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df\": container with ID starting with fec6f355c70116fc7d7c76ddc817c01cb43d46f48a41ad9ffc2d5b436bf649df not found: ID does not exist"
Sep 18 21:09:40.712223 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:40.712189    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-sv9qx"]
Sep 18 21:09:41.460220 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.460192    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" path="/var/lib/kubelet/pods/6dc74968-f6bb-4ff4-b79b-26b8982b0247/volumes"
Sep 18 21:09:41.829567 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.829542    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:09:41.903401 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.903374    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-cabundle-cert\") pod \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") "
Sep 18 21:09:41.903555 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.903529    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kserve-provision-location\") pod \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") "
Sep 18 21:09:41.903625 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.903575    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqzw4\" (UniqueName: \"kubernetes.io/projected/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-api-access-fqzw4\") pod \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") "
Sep 18 21:09:41.903625 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.903612    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-proxy-tls\") pod \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") "
Sep 18 21:09:41.903729 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.903641    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-rbac-proxy-sar-config\") pod \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\" (UID: \"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6\") "
Sep 18 21:09:41.903729 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.903681    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" (UID: "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:09:41.903868 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.903853    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:41.904078 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.904048    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" (UID: "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:09:41.905600 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.905581    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-api-access-fqzw4" (OuterVolumeSpecName: "kube-api-access-fqzw4") pod "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" (UID: "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6"). InnerVolumeSpecName "kube-api-access-fqzw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:09:41.905712 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.905695    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" (UID: "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:09:41.913825 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:41.913804    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" (UID: "c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:09:42.005028 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.004934    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqzw4\" (UniqueName: \"kubernetes.io/projected/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-api-access-fqzw4\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:42.005028 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.004967    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:42.005028 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.004977    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:42.005028 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.004986    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:09:42.697346 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.697318    2566 generic.go:356] "Generic (PLEG): container finished" podID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerID="d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a" exitCode=0
Sep 18 21:09:42.697804 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.697367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" event={"ID":"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6","Type":"ContainerDied","Data":"d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a"}
Sep 18 21:09:42.697804 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.697382    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"
Sep 18 21:09:42.697804 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.697427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n" event={"ID":"c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6","Type":"ContainerDied","Data":"2f53b6419126374c94fa9754e0ff16453bfb51935304030455a37d6ae95422f1"}
Sep 18 21:09:42.697804 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.697466    2566 scope.go:117] "RemoveContainer" containerID="281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090"
Sep 18 21:09:42.709560 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.709243    2566 scope.go:117] "RemoveContainer" containerID="d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a"
Sep 18 21:09:42.717154 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.717136    2566 scope.go:117] "RemoveContainer" containerID="096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778"
Sep 18 21:09:42.724104 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.724080    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"]
Sep 18 21:09:42.724314 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.724298    2566 scope.go:117] "RemoveContainer" containerID="281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090"
Sep 18 21:09:42.724567 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:09:42.724551    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090\": container with ID starting with 281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090 not found: ID does not exist" containerID="281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090"
Sep 18 21:09:42.724638 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.724577    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090"} err="failed to get container status \"281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090\": rpc error: code = NotFound desc = could not find container \"281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090\": container with ID starting with 281092cb4b35eceae92e300d9e80bd87e13392a27718887bd6877c6930db3090 not found: ID does not exist"
Sep 18 21:09:42.724638 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.724599    2566 scope.go:117] "RemoveContainer" containerID="d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a"
Sep 18 21:09:42.724835 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:09:42.724819    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a\": container with ID starting with d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a not found: ID does not exist" containerID="d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a"
Sep 18 21:09:42.724874 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.724848    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a"} err="failed to get container status \"d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a\": rpc error: code = NotFound desc = could not find container \"d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a\": container with ID starting with d32bccbd6e7be3f522ce129be60c6a516b0d5c20a14b20628253857b16b8ae4a not found: ID does not exist"
Sep 18 21:09:42.724874 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.724863    2566 scope.go:117] "RemoveContainer" containerID="096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778"
Sep 18 21:09:42.725130 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:09:42.725115    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778\": container with ID starting with 096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778 not found: ID does not exist" containerID="096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778"
Sep 18 21:09:42.725185 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.725132    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778"} err="failed to get container status \"096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778\": rpc error: code = NotFound desc = could not find container \"096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778\": container with ID starting with 096aac701565b35489d21488b45168fbf25ae37089667c847dc723498b80e778 not found: ID does not exist"
Sep 18 21:09:42.727937 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:42.727917    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-kr99n"]
Sep 18 21:09:43.460096 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:43.460064    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" path="/var/lib/kubelet/pods/c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6/volumes"
Sep 18 21:09:44.091306 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091276    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"]
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091548    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091559    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091568    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="storage-initializer"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091576    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="storage-initializer"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091592    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="storage-initializer"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091598    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="storage-initializer"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091606    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kube-rbac-proxy"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091610    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kube-rbac-proxy"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091615    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kube-rbac-proxy"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091620    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kube-rbac-proxy"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091629    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091635    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container"
Sep 18 21:09:44.091669 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091676    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kserve-container"
Sep 18 21:09:44.092096 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091684    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kube-rbac-proxy"
Sep 18 21:09:44.092096 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091690    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d105f6-86ec-4572-a6ec-ab46e9d2f0b6" containerName="kserve-container"
Sep 18 21:09:44.092096 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.091697    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc74968-f6bb-4ff4-b79b-26b8982b0247" containerName="kube-rbac-proxy"
Sep 18 21:09:44.096103 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.096088    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.098875 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.098852    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 18 21:09:44.099145 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.099129    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 21:09:44.099266 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.099186    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 21:09:44.099266 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.099186    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 18 21:09:44.099266 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.099228    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sgzg4\""
Sep 18 21:09:44.100133 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.100118    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 21:09:44.100211 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.100146    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 21:09:44.105672 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.105649    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"]
Sep 18 21:09:44.120686 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.120660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65n6x\" (UniqueName: \"kubernetes.io/projected/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-api-access-65n6x\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.120788 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.120715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.120788 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.120776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.120874 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.120833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.120874 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.120854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.221555 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.221525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.221693 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.221564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.221693 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.221587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.221693 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.221603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.221693 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.221624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65n6x\" (UniqueName: \"kubernetes.io/projected/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-api-access-65n6x\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.222003 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.221981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.222348 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.222329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.222420 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.222333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.223999 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.223980    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.229932 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.229912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65n6x\" (UniqueName: \"kubernetes.io/projected/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-api-access-65n6x\") pod \"isvc-pmml-predictor-8674795f59-d9mcm\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.407973 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.407896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:44.677377 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.677353    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"]
Sep 18 21:09:44.679401 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:09:44.679371    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acd5cd7_c300_42c0_b06c_aff28fca8f2c.slice/crio-c59ff69a06d38366d15768ce146016bbbc1df22c060c5c37936ed76e6515eb22 WatchSource:0}: Error finding container c59ff69a06d38366d15768ce146016bbbc1df22c060c5c37936ed76e6515eb22: Status 404 returned error can't find the container with id c59ff69a06d38366d15768ce146016bbbc1df22c060c5c37936ed76e6515eb22
Sep 18 21:09:44.707410 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:44.707379    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" event={"ID":"6acd5cd7-c300-42c0-b06c-aff28fca8f2c","Type":"ContainerStarted","Data":"c59ff69a06d38366d15768ce146016bbbc1df22c060c5c37936ed76e6515eb22"}
Sep 18 21:09:45.712776 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:45.712744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" event={"ID":"6acd5cd7-c300-42c0-b06c-aff28fca8f2c","Type":"ContainerStarted","Data":"ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7"}
Sep 18 21:09:47.415638 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.415599    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"]
Sep 18 21:09:47.419228 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.419206    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.421863 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.421844    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 18 21:09:47.421982 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.421911    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 18 21:09:47.429311 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.429290    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"]
Sep 18 21:09:47.442891 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.442866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b096e8-c8b5-4761-907d-fabcbabda4fb-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.442968 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.442917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b096e8-c8b5-4761-907d-fabcbabda4fb-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.442968 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.442938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9vz\" (UniqueName: \"kubernetes.io/projected/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-api-access-sl9vz\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.442968 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.442959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.443097 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.443004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.543708 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.543674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.543848 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.543719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b096e8-c8b5-4761-907d-fabcbabda4fb-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.543848 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.543756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b096e8-c8b5-4761-907d-fabcbabda4fb-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.543848 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.543774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9vz\" (UniqueName: \"kubernetes.io/projected/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-api-access-sl9vz\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.543848 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.543792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.544262 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.544239    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b096e8-c8b5-4761-907d-fabcbabda4fb-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.544447 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.544431    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.544500 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.544483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.546279 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.546257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b096e8-c8b5-4761-907d-fabcbabda4fb-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.552089 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.552066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9vz\" (UniqueName: \"kubernetes.io/projected/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-api-access-sl9vz\") pod \"isvc-pmml-runtime-predictor-885fc8777-k64tt\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:47.731067 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:47.730970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:48.001612 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:48.001589    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"]
Sep 18 21:09:48.003935 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:09:48.003908    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b096e8_c8b5_4761_907d_fabcbabda4fb.slice/crio-e610243a89789eff9dc0d2b97a895f2d324a4d8961aa5bf42b59254ec66ea5f1 WatchSource:0}: Error finding container e610243a89789eff9dc0d2b97a895f2d324a4d8961aa5bf42b59254ec66ea5f1: Status 404 returned error can't find the container with id e610243a89789eff9dc0d2b97a895f2d324a4d8961aa5bf42b59254ec66ea5f1
Sep 18 21:09:48.725324 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:48.725291    2566 generic.go:356] "Generic (PLEG): container finished" podID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerID="ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7" exitCode=0
Sep 18 21:09:48.725740 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:48.725364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" event={"ID":"6acd5cd7-c300-42c0-b06c-aff28fca8f2c","Type":"ContainerDied","Data":"ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7"}
Sep 18 21:09:48.726933 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:48.726908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" event={"ID":"80b096e8-c8b5-4761-907d-fabcbabda4fb","Type":"ContainerStarted","Data":"d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea"}
Sep 18 21:09:48.727052 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:48.726966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" event={"ID":"80b096e8-c8b5-4761-907d-fabcbabda4fb","Type":"ContainerStarted","Data":"e610243a89789eff9dc0d2b97a895f2d324a4d8961aa5bf42b59254ec66ea5f1"}
Sep 18 21:09:53.759590 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:53.759547    2566 generic.go:356] "Generic (PLEG): container finished" podID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerID="d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea" exitCode=0
Sep 18 21:09:53.760002 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:53.759638    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" event={"ID":"80b096e8-c8b5-4761-907d-fabcbabda4fb","Type":"ContainerDied","Data":"d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea"}
Sep 18 21:09:56.775388 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:56.775352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" event={"ID":"6acd5cd7-c300-42c0-b06c-aff28fca8f2c","Type":"ContainerStarted","Data":"f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e"}
Sep 18 21:09:56.775830 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:56.775424    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" event={"ID":"6acd5cd7-c300-42c0-b06c-aff28fca8f2c","Type":"ContainerStarted","Data":"66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea"}
Sep 18 21:09:56.775830 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:56.775620    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:56.777490 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:56.777467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" event={"ID":"80b096e8-c8b5-4761-907d-fabcbabda4fb","Type":"ContainerStarted","Data":"b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291"}
Sep 18 21:09:56.777628 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:56.777523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" event={"ID":"80b096e8-c8b5-4761-907d-fabcbabda4fb","Type":"ContainerStarted","Data":"dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45"}
Sep 18 21:09:56.777719 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:56.777701    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:56.793369 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:56.793284    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podStartSLOduration=1.941468028 podStartE2EDuration="12.793267533s" totalImagesPullingTime="7.36643866s" totalInitContainerRuntime="3.485360845s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:09:56.792438816 +0000 UTC m=+2965.823364968" watchObservedRunningTime="2026-09-18 21:09:56.793267533 +0000 UTC m=+2965.824193674"
Sep 18 21:09:56.810190 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:56.810144    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podStartSLOduration=3.597517188 podStartE2EDuration="9.810133824s" totalImagesPullingTime="2.307007531s" totalInitContainerRuntime="3.905609105s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:09:56.80912256 +0000 UTC m=+2965.840048711" watchObservedRunningTime="2026-09-18 21:09:56.810133824 +0000 UTC m=+2965.841059964"
Sep 18 21:09:57.782767 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:57.782736    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:09:57.783183 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:57.782797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:09:57.783854 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:57.783828    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:09:57.783945 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:57.783850    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:09:58.787384 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:58.787344    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:09:58.787775 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:09:58.787428    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:10:03.791741 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:03.791711    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:10:03.792766 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:03.792741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:10:03.792902 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:03.792786    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:10:03.793045 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:03.793002    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:10:13.792670 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:13.792629    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:10:13.793163 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:13.793065    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:10:23.792705 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:23.792668    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:10:23.793119 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:23.793039    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:10:31.586645 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:31.586616    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 21:10:31.592247 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:31.592229    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 21:10:33.793658 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:33.793620    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:10:33.794054 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:33.793653    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:10:43.793323 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:43.793283    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:10:43.793732 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:43.793288    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:10:53.793463 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:53.793424    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:10:53.793928 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:10:53.793423    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:11:03.793399 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:03.793359    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:11:03.793793 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:03.793412    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:11:13.793561 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:13.793523    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:11:13.794056 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:13.793521    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 18 21:11:15.455954 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:15.455906    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:11:23.793776 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:23.793746    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:11:25.459199 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:25.459173    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:11:25.812533 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:25.812492    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"]
Sep 18 21:11:25.812725 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:25.812701    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container" containerID="cri-o://66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea" gracePeriod=30
Sep 18 21:11:25.812793 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:25.812746    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kube-rbac-proxy" containerID="cri-o://f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e" gracePeriod=30
Sep 18 21:11:26.139153 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:26.139069    2566 generic.go:356] "Generic (PLEG): container finished" podID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerID="f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e" exitCode=2
Sep 18 21:11:26.139153 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:26.139132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" event={"ID":"6acd5cd7-c300-42c0-b06c-aff28fca8f2c","Type":"ContainerDied","Data":"f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e"}
Sep 18 21:11:28.787745 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:28.787696    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Sep 18 21:11:29.751668 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.751643    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:11:29.877658 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.877575    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-cabundle-cert\") pod \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") "
Sep 18 21:11:29.877658 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.877631    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-rbac-proxy-sar-config\") pod \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") "
Sep 18 21:11:29.878185 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.877673    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-proxy-tls\") pod \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") "
Sep 18 21:11:29.878185 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.877702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65n6x\" (UniqueName: \"kubernetes.io/projected/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-api-access-65n6x\") pod \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") "
Sep 18 21:11:29.878185 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.877743    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kserve-provision-location\") pod \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\" (UID: \"6acd5cd7-c300-42c0-b06c-aff28fca8f2c\") "
Sep 18 21:11:29.878185 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.878065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6acd5cd7-c300-42c0-b06c-aff28fca8f2c" (UID: "6acd5cd7-c300-42c0-b06c-aff28fca8f2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:11:29.878185 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.878097    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6acd5cd7-c300-42c0-b06c-aff28fca8f2c" (UID: "6acd5cd7-c300-42c0-b06c-aff28fca8f2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:11:29.878185 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.878123    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6acd5cd7-c300-42c0-b06c-aff28fca8f2c" (UID: "6acd5cd7-c300-42c0-b06c-aff28fca8f2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:11:29.879745 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.879722    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-api-access-65n6x" (OuterVolumeSpecName: "kube-api-access-65n6x") pod "6acd5cd7-c300-42c0-b06c-aff28fca8f2c" (UID: "6acd5cd7-c300-42c0-b06c-aff28fca8f2c"). InnerVolumeSpecName "kube-api-access-65n6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:11:29.879837 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.879790    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6acd5cd7-c300-42c0-b06c-aff28fca8f2c" (UID: "6acd5cd7-c300-42c0-b06c-aff28fca8f2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:11:29.978527 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.978497    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:29.978527 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.978521    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:29.978527 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.978531    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:29.978720 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.978540    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65n6x\" (UniqueName: \"kubernetes.io/projected/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kube-api-access-65n6x\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:29.978720 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:29.978548    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6acd5cd7-c300-42c0-b06c-aff28fca8f2c-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:30.155870 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.155781    2566 generic.go:356] "Generic (PLEG): container finished" podID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerID="66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea" exitCode=0
Sep 18 21:11:30.155870 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.155837    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"
Sep 18 21:11:30.155870 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.155848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" event={"ID":"6acd5cd7-c300-42c0-b06c-aff28fca8f2c","Type":"ContainerDied","Data":"66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea"}
Sep 18 21:11:30.156171 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.155910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm" event={"ID":"6acd5cd7-c300-42c0-b06c-aff28fca8f2c","Type":"ContainerDied","Data":"c59ff69a06d38366d15768ce146016bbbc1df22c060c5c37936ed76e6515eb22"}
Sep 18 21:11:30.156171 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.155947    2566 scope.go:117] "RemoveContainer" containerID="f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e"
Sep 18 21:11:30.164776 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.164759    2566 scope.go:117] "RemoveContainer" containerID="66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea"
Sep 18 21:11:30.171925 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.171906    2566 scope.go:117] "RemoveContainer" containerID="ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7"
Sep 18 21:11:30.179151 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.179128    2566 scope.go:117] "RemoveContainer" containerID="f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e"
Sep 18 21:11:30.179483 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:11:30.179442    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e\": container with ID starting with f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e not found: ID does not exist" containerID="f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e"
Sep 18 21:11:30.179483 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.179474    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e"} err="failed to get container status \"f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e\": rpc error: code = NotFound desc = could not find container \"f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e\": container with ID starting with f2ef81b0c710ad0b9235c4dd7649038655e4d47a19877e0b522755d49be8bf6e not found: ID does not exist"
Sep 18 21:11:30.179630 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.179490    2566 scope.go:117] "RemoveContainer" containerID="66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea"
Sep 18 21:11:30.179776 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:11:30.179759    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea\": container with ID starting with 66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea not found: ID does not exist" containerID="66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea"
Sep 18 21:11:30.179840 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.179781    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea"} err="failed to get container status \"66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea\": rpc error: code = NotFound desc = could not find container \"66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea\": container with ID starting with 66e7591904e2b71325b1f284c23af8ab92c12217813250f603ebc899dc711cea not found: ID does not exist"
Sep 18 21:11:30.179840 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.179795    2566 scope.go:117] "RemoveContainer" containerID="ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7"
Sep 18 21:11:30.180052 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:11:30.180032    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7\": container with ID starting with ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7 not found: ID does not exist" containerID="ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7"
Sep 18 21:11:30.180118 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.180057    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7"} err="failed to get container status \"ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7\": rpc error: code = NotFound desc = could not find container \"ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7\": container with ID starting with ca2e6db1ff552fc80681c09da8dee10816fde02448f22ae7119ea91a69b87cd7 not found: ID does not exist"
Sep 18 21:11:30.180248 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.180230    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"]
Sep 18 21:11:30.183326 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:30.183306    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-d9mcm"]
Sep 18 21:11:31.003487 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.003446    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"]
Sep 18 21:11:31.003850 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.003763    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="storage-initializer"
Sep 18 21:11:31.003850 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.003773    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="storage-initializer"
Sep 18 21:11:31.003850 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.003783    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container"
Sep 18 21:11:31.003850 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.003788    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container"
Sep 18 21:11:31.003850 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.003798    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kube-rbac-proxy"
Sep 18 21:11:31.003850 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.003804    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kube-rbac-proxy"
Sep 18 21:11:31.004070 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.003863    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kserve-container"
Sep 18 21:11:31.004070 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.003872    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" containerName="kube-rbac-proxy"
Sep 18 21:11:31.008314 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.008297    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.010851 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.010831    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-51789d-predictor-serving-cert\""
Sep 18 21:11:31.010942 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.010881    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-51789d-kube-rbac-proxy-sar-config\""
Sep 18 21:11:31.017469 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.017449    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"]
Sep 18 21:11:31.188450 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.188421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-cabundle-cert\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.188656 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.188463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/897f4c50-f820-43e9-971f-11836288f774-kube-api-access-6mpkt\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.188656 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.188483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-kube-rbac-proxy-sar-config\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.188656 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.188582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897f4c50-f820-43e9-971f-11836288f774-kserve-provision-location\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.188793 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.188670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897f4c50-f820-43e9-971f-11836288f774-proxy-tls\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.289577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.289540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/897f4c50-f820-43e9-971f-11836288f774-kube-api-access-6mpkt\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.289577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.289584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-kube-rbac-proxy-sar-config\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.289818 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.289797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897f4c50-f820-43e9-971f-11836288f774-kserve-provision-location\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.289901 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.289885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897f4c50-f820-43e9-971f-11836288f774-proxy-tls\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.289942 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.289923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-cabundle-cert\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.290198 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.290181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-kube-rbac-proxy-sar-config\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.290238 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.290196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897f4c50-f820-43e9-971f-11836288f774-kserve-provision-location\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.290469 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.290443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-cabundle-cert\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.292298 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.292280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897f4c50-f820-43e9-971f-11836288f774-proxy-tls\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.298421 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.298397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/897f4c50-f820-43e9-971f-11836288f774-kube-api-access-6mpkt\") pod \"isvc-primary-51789d-predictor-7659f869cf-7nrf5\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") " pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.321364 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.321342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:31.459813 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.459785    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6acd5cd7-c300-42c0-b06c-aff28fca8f2c" path="/var/lib/kubelet/pods/6acd5cd7-c300-42c0-b06c-aff28fca8f2c/volumes"
Sep 18 21:11:31.595340 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:31.595316    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"]
Sep 18 21:11:31.597051 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:11:31.597008    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897f4c50_f820_43e9_971f_11836288f774.slice/crio-db9ef66e6785d4ace252e4f2e10ce84078ce13f66bfb4e9e0a6fa39ad1894ce7 WatchSource:0}: Error finding container db9ef66e6785d4ace252e4f2e10ce84078ce13f66bfb4e9e0a6fa39ad1894ce7: Status 404 returned error can't find the container with id db9ef66e6785d4ace252e4f2e10ce84078ce13f66bfb4e9e0a6fa39ad1894ce7
Sep 18 21:11:32.166450 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:32.166414    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" event={"ID":"897f4c50-f820-43e9-971f-11836288f774","Type":"ContainerStarted","Data":"d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35"}
Sep 18 21:11:32.166873 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:32.166470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" event={"ID":"897f4c50-f820-43e9-971f-11836288f774","Type":"ContainerStarted","Data":"db9ef66e6785d4ace252e4f2e10ce84078ce13f66bfb4e9e0a6fa39ad1894ce7"}
Sep 18 21:11:33.279283 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:33.279253    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"]
Sep 18 21:11:33.279659 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:33.279486    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" containerID="cri-o://dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45" gracePeriod=30
Sep 18 21:11:33.279659 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:33.279561    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kube-rbac-proxy" containerID="cri-o://b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291" gracePeriod=30
Sep 18 21:11:33.788307 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:33.788262    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Sep 18 21:11:34.177606 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:34.177514    2566 generic.go:356] "Generic (PLEG): container finished" podID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerID="b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291" exitCode=2
Sep 18 21:11:34.177606 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:34.177585    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" event={"ID":"80b096e8-c8b5-4761-907d-fabcbabda4fb","Type":"ContainerDied","Data":"b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291"}
Sep 18 21:11:35.456345 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:35.456297    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 18 21:11:36.186822 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:36.186795    2566 generic.go:356] "Generic (PLEG): container finished" podID="897f4c50-f820-43e9-971f-11836288f774" containerID="d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35" exitCode=0
Sep 18 21:11:36.186999 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:36.186875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" event={"ID":"897f4c50-f820-43e9-971f-11836288f774","Type":"ContainerDied","Data":"d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35"}
Sep 18 21:11:37.193841 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.193813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" event={"ID":"897f4c50-f820-43e9-971f-11836288f774","Type":"ContainerStarted","Data":"56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d"}
Sep 18 21:11:37.194210 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.193872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" event={"ID":"897f4c50-f820-43e9-971f-11836288f774","Type":"ContainerStarted","Data":"9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9"}
Sep 18 21:11:37.194210 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.194130    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:37.213050 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.212988    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podStartSLOduration=3.491711272 podStartE2EDuration="7.212971623s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721260351s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:37.211093963 +0000 UTC m=+3066.242020104" watchObservedRunningTime="2026-09-18 21:11:37.212971623 +0000 UTC m=+3066.243897763"
Sep 18 21:11:37.319823 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.319799    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:11:37.339556 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.339530    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b096e8-c8b5-4761-907d-fabcbabda4fb-kserve-provision-location\") pod \"80b096e8-c8b5-4761-907d-fabcbabda4fb\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") "
Sep 18 21:11:37.339691 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.339572    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-cabundle-cert\") pod \"80b096e8-c8b5-4761-907d-fabcbabda4fb\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") "
Sep 18 21:11:37.339691 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.339603    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b096e8-c8b5-4761-907d-fabcbabda4fb-proxy-tls\") pod \"80b096e8-c8b5-4761-907d-fabcbabda4fb\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") "
Sep 18 21:11:37.339691 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.339637    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl9vz\" (UniqueName: \"kubernetes.io/projected/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-api-access-sl9vz\") pod \"80b096e8-c8b5-4761-907d-fabcbabda4fb\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") "
Sep 18 21:11:37.339871 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.339695    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-rbac-proxy-sar-config\") pod \"80b096e8-c8b5-4761-907d-fabcbabda4fb\" (UID: \"80b096e8-c8b5-4761-907d-fabcbabda4fb\") "
Sep 18 21:11:37.339931 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.339885    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b096e8-c8b5-4761-907d-fabcbabda4fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80b096e8-c8b5-4761-907d-fabcbabda4fb" (UID: "80b096e8-c8b5-4761-907d-fabcbabda4fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:11:37.339931 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.339891    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80b096e8-c8b5-4761-907d-fabcbabda4fb" (UID: "80b096e8-c8b5-4761-907d-fabcbabda4fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:11:37.340053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.339961    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:37.340053 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.339977    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b096e8-c8b5-4761-907d-fabcbabda4fb-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:37.340324 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.340291    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80b096e8-c8b5-4761-907d-fabcbabda4fb" (UID: "80b096e8-c8b5-4761-907d-fabcbabda4fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:11:37.342154 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.342115    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-api-access-sl9vz" (OuterVolumeSpecName: "kube-api-access-sl9vz") pod "80b096e8-c8b5-4761-907d-fabcbabda4fb" (UID: "80b096e8-c8b5-4761-907d-fabcbabda4fb"). InnerVolumeSpecName "kube-api-access-sl9vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:11:37.342253 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.342188    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b096e8-c8b5-4761-907d-fabcbabda4fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80b096e8-c8b5-4761-907d-fabcbabda4fb" (UID: "80b096e8-c8b5-4761-907d-fabcbabda4fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:11:37.441305 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.441241    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:37.441305 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.441265    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b096e8-c8b5-4761-907d-fabcbabda4fb-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:37.441305 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:37.441275    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl9vz\" (UniqueName: \"kubernetes.io/projected/80b096e8-c8b5-4761-907d-fabcbabda4fb-kube-api-access-sl9vz\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:11:38.199393 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.199358    2566 generic.go:356] "Generic (PLEG): container finished" podID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerID="dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45" exitCode=0
Sep 18 21:11:38.199837 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.199418    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"
Sep 18 21:11:38.199837 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.199418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" event={"ID":"80b096e8-c8b5-4761-907d-fabcbabda4fb","Type":"ContainerDied","Data":"dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45"}
Sep 18 21:11:38.199837 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.199479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt" event={"ID":"80b096e8-c8b5-4761-907d-fabcbabda4fb","Type":"ContainerDied","Data":"e610243a89789eff9dc0d2b97a895f2d324a4d8961aa5bf42b59254ec66ea5f1"}
Sep 18 21:11:38.199837 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.199505    2566 scope.go:117] "RemoveContainer" containerID="b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291"
Sep 18 21:11:38.200065 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.200048    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:38.201780 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.201754    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 18 21:11:38.207662 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.207647    2566 scope.go:117] "RemoveContainer" containerID="dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45"
Sep 18 21:11:38.215200 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.214984    2566 scope.go:117] "RemoveContainer" containerID="d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea"
Sep 18 21:11:38.216947 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.216931    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"]
Sep 18 21:11:38.220251 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.220232    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k64tt"]
Sep 18 21:11:38.222416 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.222402    2566 scope.go:117] "RemoveContainer" containerID="b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291"
Sep 18 21:11:38.222678 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:11:38.222661    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291\": container with ID starting with b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291 not found: ID does not exist" containerID="b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291"
Sep 18 21:11:38.222746 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.222691    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291"} err="failed to get container status \"b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291\": rpc error: code = NotFound desc = could not find container \"b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291\": container with ID starting with b9270544dfc5f90105c946bf485906930e19e29366aac8c90d37c4612335d291 not found: ID does not exist"
Sep 18 21:11:38.222746 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.222716    2566 scope.go:117] "RemoveContainer" containerID="dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45"
Sep 18 21:11:38.222957 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:11:38.222941    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45\": container with ID starting with dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45 not found: ID does not exist" containerID="dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45"
Sep 18 21:11:38.222998 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.222963    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45"} err="failed to get container status \"dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45\": rpc error: code = NotFound desc = could not find container \"dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45\": container with ID starting with dff5d14a54bf906b87f6947f0b6625b62dd64d3ccf5d95b1db33f1062ed8ea45 not found: ID does not exist"
Sep 18 21:11:38.222998 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.222978    2566 scope.go:117] "RemoveContainer" containerID="d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea"
Sep 18 21:11:38.223237 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:11:38.223223    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea\": container with ID starting with d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea not found: ID does not exist" containerID="d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea"
Sep 18 21:11:38.223283 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.223242    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea"} err="failed to get container status \"d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea\": rpc error: code = NotFound desc = could not find container \"d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea\": container with ID starting with d17d42b85368fc4dcb4245b47c4cc4a5811a8b8f9c883066788939c7c1b221ea not found: ID does not exist"
Sep 18 21:11:38.524875 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.524843    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"]
Sep 18 21:11:38.525147 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.525134    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="storage-initializer"
Sep 18 21:11:38.525206 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.525148    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="storage-initializer"
Sep 18 21:11:38.525206 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.525161    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kube-rbac-proxy"
Sep 18 21:11:38.525206 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.525166    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kube-rbac-proxy"
Sep 18 21:11:38.525206 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.525175    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container"
Sep 18 21:11:38.525206 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.525182    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container"
Sep 18 21:11:38.525366 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.525235    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kube-rbac-proxy"
Sep 18 21:11:38.525366 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.525243    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" containerName="kserve-container"
Sep 18 21:11:38.529389 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.529370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.531751 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.531727    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 18 21:11:38.531855 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.531802    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 21:11:38.537927 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.537904    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"]
Sep 18 21:11:38.549407 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.549370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b49dc5d7-6299-4faa-9a14-331756d813ff-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.549735 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.549704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.549838 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.549804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.549907 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.549837    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6j8s\" (UniqueName: \"kubernetes.io/projected/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-api-access-w6j8s\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.549907 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.549863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b49dc5d7-6299-4faa-9a14-331756d813ff-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.650343 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.650314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6j8s\" (UniqueName: \"kubernetes.io/projected/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-api-access-w6j8s\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.650470 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.650347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b49dc5d7-6299-4faa-9a14-331756d813ff-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.650470 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.650382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b49dc5d7-6299-4faa-9a14-331756d813ff-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.650470 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.650437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.650709 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.650493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.650709 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:11:38.650532    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 18 21:11:38.650709 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:11:38.650635    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b49dc5d7-6299-4faa-9a14-331756d813ff-proxy-tls podName:b49dc5d7-6299-4faa-9a14-331756d813ff nodeName:}" failed. No retries permitted until 2026-09-18 21:11:39.150611959 +0000 UTC m=+3068.181538093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b49dc5d7-6299-4faa-9a14-331756d813ff-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" (UID: "b49dc5d7-6299-4faa-9a14-331756d813ff") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 18 21:11:38.650880 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.650729    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b49dc5d7-6299-4faa-9a14-331756d813ff-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.651200 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.651178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.651285 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.651222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:38.658805 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:38.658780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6j8s\" (UniqueName: \"kubernetes.io/projected/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-api-access-w6j8s\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:39.153308 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:39.153271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b49dc5d7-6299-4faa-9a14-331756d813ff-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:39.155711 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:39.155689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b49dc5d7-6299-4faa-9a14-331756d813ff-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:39.204322 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:39.204281    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 18 21:11:39.439930 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:39.439832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:39.460576 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:39.460549    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80b096e8-c8b5-4761-907d-fabcbabda4fb" path="/var/lib/kubelet/pods/80b096e8-c8b5-4761-907d-fabcbabda4fb/volumes"
Sep 18 21:11:39.710060 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:39.710034    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"]
Sep 18 21:11:39.711931 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:11:39.711905    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49dc5d7_6299_4faa_9a14_331756d813ff.slice/crio-f728cae3abef5f629479a4ed5e3f4ab143fbb517d7980cce90e0f7141f508895 WatchSource:0}: Error finding container f728cae3abef5f629479a4ed5e3f4ab143fbb517d7980cce90e0f7141f508895: Status 404 returned error can't find the container with id f728cae3abef5f629479a4ed5e3f4ab143fbb517d7980cce90e0f7141f508895
Sep 18 21:11:40.209202 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:40.209163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" event={"ID":"b49dc5d7-6299-4faa-9a14-331756d813ff","Type":"ContainerStarted","Data":"1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab"}
Sep 18 21:11:40.209582 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:40.209231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" event={"ID":"b49dc5d7-6299-4faa-9a14-331756d813ff","Type":"ContainerStarted","Data":"f728cae3abef5f629479a4ed5e3f4ab143fbb517d7980cce90e0f7141f508895"}
Sep 18 21:11:44.208431 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:44.208402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:11:44.209041 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:44.208998    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 18 21:11:44.225584 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:44.225561    2566 generic.go:356] "Generic (PLEG): container finished" podID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerID="1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab" exitCode=0
Sep 18 21:11:44.225707 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:44.225635    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" event={"ID":"b49dc5d7-6299-4faa-9a14-331756d813ff","Type":"ContainerDied","Data":"1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab"}
Sep 18 21:11:45.232156 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:45.232118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" event={"ID":"b49dc5d7-6299-4faa-9a14-331756d813ff","Type":"ContainerStarted","Data":"c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c"}
Sep 18 21:11:45.232523 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:45.232212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" event={"ID":"b49dc5d7-6299-4faa-9a14-331756d813ff","Type":"ContainerStarted","Data":"73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5"}
Sep 18 21:11:45.232523 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:45.232374    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:45.252777 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:45.252732    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podStartSLOduration=3.311731555 podStartE2EDuration="7.252719001s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.940987446s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:45.250317338 +0000 UTC m=+3074.281243478" watchObservedRunningTime="2026-09-18 21:11:45.252719001 +0000 UTC m=+3074.283645140"
Sep 18 21:11:46.236482 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:46.236450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:46.237794 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:46.237762    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:11:47.241497 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:47.241461    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:11:52.245677 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:52.245648    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:11:52.246189 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:52.246159    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:11:54.208948 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:11:54.208913    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 18 21:12:02.246447 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:02.246405    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:12:04.209646 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:04.209609    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 18 21:12:12.246298 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:12.246262    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:12:14.209172 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:14.209129    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 18 21:12:22.246737 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:22.246697    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:12:24.209657 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:24.209615    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 18 21:12:32.246788 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:32.246687    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:12:34.208963 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:34.208928    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 18 21:12:42.246437 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:42.246402    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:12:44.209636 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:44.209591    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 18 21:12:52.246342 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:52.246308    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:12:54.210113 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:12:54.210034    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:13:02.246810 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:02.246774    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:13:12.247214 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:12.247181    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:13:16.213647 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.213610    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"]
Sep 18 21:13:16.216899 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.216883    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.219511 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.219476    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-51789d\""
Sep 18 21:13:16.219650 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.219520    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-51789d-predictor-serving-cert\""
Sep 18 21:13:16.219650 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.219520    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-51789d-kube-rbac-proxy-sar-config\""
Sep 18 21:13:16.219650 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.219520    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-51789d-dockercfg-8vt4q\""
Sep 18 21:13:16.223298 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.223275    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"]
Sep 18 21:13:16.259283 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.259260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/520b3434-38c2-40f4-a837-1a892aa5455a-proxy-tls\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.259397 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.259298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxj46\" (UniqueName: \"kubernetes.io/projected/520b3434-38c2-40f4-a837-1a892aa5455a-kube-api-access-kxj46\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.259397 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.259321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/520b3434-38c2-40f4-a837-1a892aa5455a-kserve-provision-location\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.259397 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.259360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-cabundle-cert\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.259517 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.259422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.360457 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.360429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxj46\" (UniqueName: \"kubernetes.io/projected/520b3434-38c2-40f4-a837-1a892aa5455a-kube-api-access-kxj46\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.360579 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.360468    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/520b3434-38c2-40f4-a837-1a892aa5455a-kserve-provision-location\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.360579 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.360497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-cabundle-cert\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.360579 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.360548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.360725 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.360610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/520b3434-38c2-40f4-a837-1a892aa5455a-proxy-tls\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.360725 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:16.360716    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-51789d-predictor-serving-cert: secret "isvc-secondary-51789d-predictor-serving-cert" not found
Sep 18 21:13:16.360835 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:16.360781    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/520b3434-38c2-40f4-a837-1a892aa5455a-proxy-tls podName:520b3434-38c2-40f4-a837-1a892aa5455a nodeName:}" failed. No retries permitted until 2026-09-18 21:13:16.860761389 +0000 UTC m=+3165.891687510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/520b3434-38c2-40f4-a837-1a892aa5455a-proxy-tls") pod "isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp" (UID: "520b3434-38c2-40f4-a837-1a892aa5455a") : secret "isvc-secondary-51789d-predictor-serving-cert" not found
Sep 18 21:13:16.360904 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.360846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/520b3434-38c2-40f4-a837-1a892aa5455a-kserve-provision-location\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.361214 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.361194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-cabundle-cert\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.361303 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.361266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.371464 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.371436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxj46\" (UniqueName: \"kubernetes.io/projected/520b3434-38c2-40f4-a837-1a892aa5455a-kube-api-access-kxj46\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.865004 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.864965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/520b3434-38c2-40f4-a837-1a892aa5455a-proxy-tls\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:16.867405 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:16.867387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/520b3434-38c2-40f4-a837-1a892aa5455a-proxy-tls\") pod \"isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") " pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:17.128186 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:17.128113    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:17.401177 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:17.401152    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"]
Sep 18 21:13:17.403385 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:13:17.403359    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520b3434_38c2_40f4_a837_1a892aa5455a.slice/crio-5e6c723521b44f2cb86e6183c414e7935b6358c1e2a8c1af8b2eeebf2fb29df2 WatchSource:0}: Error finding container 5e6c723521b44f2cb86e6183c414e7935b6358c1e2a8c1af8b2eeebf2fb29df2: Status 404 returned error can't find the container with id 5e6c723521b44f2cb86e6183c414e7935b6358c1e2a8c1af8b2eeebf2fb29df2
Sep 18 21:13:17.598909 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:17.598871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp" event={"ID":"520b3434-38c2-40f4-a837-1a892aa5455a","Type":"ContainerStarted","Data":"49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f"}
Sep 18 21:13:17.599085 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:17.598941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp" event={"ID":"520b3434-38c2-40f4-a837-1a892aa5455a","Type":"ContainerStarted","Data":"5e6c723521b44f2cb86e6183c414e7935b6358c1e2a8c1af8b2eeebf2fb29df2"}
Sep 18 21:13:20.277996 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:20.277962    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"]
Sep 18 21:13:20.278522 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:20.278260    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" containerID="cri-o://73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5" gracePeriod=30
Sep 18 21:13:20.278522 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:20.278333    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kube-rbac-proxy" containerID="cri-o://c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c" gracePeriod=30
Sep 18 21:13:20.614743 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:20.614663    2566 generic.go:356] "Generic (PLEG): container finished" podID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerID="c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c" exitCode=2
Sep 18 21:13:20.614743 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:20.614721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" event={"ID":"b49dc5d7-6299-4faa-9a14-331756d813ff","Type":"ContainerDied","Data":"c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c"}
Sep 18 21:13:22.242257 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:22.242205    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Sep 18 21:13:22.246619 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:22.246587    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Sep 18 21:13:22.623625 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:22.623598    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp_520b3434-38c2-40f4-a837-1a892aa5455a/storage-initializer/0.log"
Sep 18 21:13:22.623793 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:22.623633    2566 generic.go:356] "Generic (PLEG): container finished" podID="520b3434-38c2-40f4-a837-1a892aa5455a" containerID="49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f" exitCode=1
Sep 18 21:13:22.623793 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:22.623676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp" event={"ID":"520b3434-38c2-40f4-a837-1a892aa5455a","Type":"ContainerDied","Data":"49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f"}
Sep 18 21:13:23.629577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:23.629551    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp_520b3434-38c2-40f4-a837-1a892aa5455a/storage-initializer/0.log"
Sep 18 21:13:23.629959 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:23.629662    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp" event={"ID":"520b3434-38c2-40f4-a837-1a892aa5455a","Type":"ContainerStarted","Data":"ddf512e703990ec1f123c96a5f22d21dcf32b369ea538b7ec5778ef5e5ea81c2"}
Sep 18 21:13:24.223313 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.223289    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:13:24.322913 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.322889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-cabundle-cert\") pod \"b49dc5d7-6299-4faa-9a14-331756d813ff\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") "
Sep 18 21:13:24.323093 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.322931    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b49dc5d7-6299-4faa-9a14-331756d813ff-proxy-tls\") pod \"b49dc5d7-6299-4faa-9a14-331756d813ff\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") "
Sep 18 21:13:24.323093 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.322960    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b49dc5d7-6299-4faa-9a14-331756d813ff-kserve-provision-location\") pod \"b49dc5d7-6299-4faa-9a14-331756d813ff\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") "
Sep 18 21:13:24.323093 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.323075    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-rbac-proxy-sar-config\") pod \"b49dc5d7-6299-4faa-9a14-331756d813ff\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") "
Sep 18 21:13:24.323258 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.323127    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6j8s\" (UniqueName: \"kubernetes.io/projected/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-api-access-w6j8s\") pod \"b49dc5d7-6299-4faa-9a14-331756d813ff\" (UID: \"b49dc5d7-6299-4faa-9a14-331756d813ff\") "
Sep 18 21:13:24.325107 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.323634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49dc5d7-6299-4faa-9a14-331756d813ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b49dc5d7-6299-4faa-9a14-331756d813ff" (UID: "b49dc5d7-6299-4faa-9a14-331756d813ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:13:24.325107 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.323686    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b49dc5d7-6299-4faa-9a14-331756d813ff" (UID: "b49dc5d7-6299-4faa-9a14-331756d813ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:13:24.325107 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.323692    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b49dc5d7-6299-4faa-9a14-331756d813ff" (UID: "b49dc5d7-6299-4faa-9a14-331756d813ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:13:24.325107 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.324008    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b49dc5d7-6299-4faa-9a14-331756d813ff-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:24.325107 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.324050    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:24.330683 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.326444    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-api-access-w6j8s" (OuterVolumeSpecName: "kube-api-access-w6j8s") pod "b49dc5d7-6299-4faa-9a14-331756d813ff" (UID: "b49dc5d7-6299-4faa-9a14-331756d813ff"). InnerVolumeSpecName "kube-api-access-w6j8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:13:24.332861 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.332838    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49dc5d7-6299-4faa-9a14-331756d813ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b49dc5d7-6299-4faa-9a14-331756d813ff" (UID: "b49dc5d7-6299-4faa-9a14-331756d813ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:13:24.425305 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.425277    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b49dc5d7-6299-4faa-9a14-331756d813ff-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:24.425305 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.425301    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b49dc5d7-6299-4faa-9a14-331756d813ff-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:24.425449 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.425310    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6j8s\" (UniqueName: \"kubernetes.io/projected/b49dc5d7-6299-4faa-9a14-331756d813ff-kube-api-access-w6j8s\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:24.636054 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.635942    2566 generic.go:356] "Generic (PLEG): container finished" podID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerID="73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5" exitCode=0
Sep 18 21:13:24.636054 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.636006    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"
Sep 18 21:13:24.636054 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.636038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" event={"ID":"b49dc5d7-6299-4faa-9a14-331756d813ff","Type":"ContainerDied","Data":"73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5"}
Sep 18 21:13:24.636604 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.636111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs" event={"ID":"b49dc5d7-6299-4faa-9a14-331756d813ff","Type":"ContainerDied","Data":"f728cae3abef5f629479a4ed5e3f4ab143fbb517d7980cce90e0f7141f508895"}
Sep 18 21:13:24.636604 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.636151    2566 scope.go:117] "RemoveContainer" containerID="c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c"
Sep 18 21:13:24.637741 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.637724    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp_520b3434-38c2-40f4-a837-1a892aa5455a/storage-initializer/1.log"
Sep 18 21:13:24.638159 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.638146    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp_520b3434-38c2-40f4-a837-1a892aa5455a/storage-initializer/0.log"
Sep 18 21:13:24.638261 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.638177    2566 generic.go:356] "Generic (PLEG): container finished" podID="520b3434-38c2-40f4-a837-1a892aa5455a" containerID="ddf512e703990ec1f123c96a5f22d21dcf32b369ea538b7ec5778ef5e5ea81c2" exitCode=1
Sep 18 21:13:24.638261 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.638213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp" event={"ID":"520b3434-38c2-40f4-a837-1a892aa5455a","Type":"ContainerDied","Data":"ddf512e703990ec1f123c96a5f22d21dcf32b369ea538b7ec5778ef5e5ea81c2"}
Sep 18 21:13:24.638550 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.638536    2566 scope.go:117] "RemoveContainer" containerID="49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f"
Sep 18 21:13:24.645617 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.645575    2566 scope.go:117] "RemoveContainer" containerID="73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5"
Sep 18 21:13:24.649532 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:24.649501    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp_kserve-ci-e2e-test(520b3434-38c2-40f4-a837-1a892aa5455a)\"" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp" podUID="520b3434-38c2-40f4-a837-1a892aa5455a"
Sep 18 21:13:24.652726 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.652706    2566 scope.go:117] "RemoveContainer" containerID="1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab"
Sep 18 21:13:24.660685 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.660669    2566 scope.go:117] "RemoveContainer" containerID="c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c"
Sep 18 21:13:24.660950 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:24.660929    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c\": container with ID starting with c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c not found: ID does not exist" containerID="c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c"
Sep 18 21:13:24.661039 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.660960    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c"} err="failed to get container status \"c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c\": rpc error: code = NotFound desc = could not find container \"c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c\": container with ID starting with c344a7670e50ccc7e80e00978f3ad299138e346fd531bd88803f3f4fc31a412c not found: ID does not exist"
Sep 18 21:13:24.661039 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.660982    2566 scope.go:117] "RemoveContainer" containerID="73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5"
Sep 18 21:13:24.661274 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:24.661257    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5\": container with ID starting with 73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5 not found: ID does not exist" containerID="73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5"
Sep 18 21:13:24.661318 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.661280    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5"} err="failed to get container status \"73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5\": rpc error: code = NotFound desc = could not find container \"73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5\": container with ID starting with 73f593ac0aad719f2485d8621ac311c6781af1f1341647d68e8af95d7b1defa5 not found: ID does not exist"
Sep 18 21:13:24.661318 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.661295    2566 scope.go:117] "RemoveContainer" containerID="1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab"
Sep 18 21:13:24.661538 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:24.661523    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab\": container with ID starting with 1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab not found: ID does not exist" containerID="1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab"
Sep 18 21:13:24.661583 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.661544    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab"} err="failed to get container status \"1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab\": rpc error: code = NotFound desc = could not find container \"1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab\": container with ID starting with 1d683e8fcb6965937741da31279cc33d4593f91e97ba537a0804d1bfd6e207ab not found: ID does not exist"
Sep 18 21:13:24.661583 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.661563    2566 scope.go:117] "RemoveContainer" containerID="49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f"
Sep 18 21:13:24.661798 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:24.661780    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f\": container with ID starting with 49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f not found: ID does not exist" containerID="49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f"
Sep 18 21:13:24.661850 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.661804    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f"} err="failed to get container status \"49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f\": rpc error: code = NotFound desc = could not find container \"49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f\": container with ID starting with 49d8f66505f7a1e9e2e9d7e525d1772145ff060c95a707e236bedd6688ce8b6f not found: ID does not exist"
Sep 18 21:13:24.669466 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.669439    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"]
Sep 18 21:13:24.673373 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:24.673354    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jbpqs"]
Sep 18 21:13:25.460025 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:25.459935    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" path="/var/lib/kubelet/pods/b49dc5d7-6299-4faa-9a14-331756d813ff/volumes"
Sep 18 21:13:25.644959 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:25.644933    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp_520b3434-38c2-40f4-a837-1a892aa5455a/storage-initializer/1.log"
Sep 18 21:13:34.217934 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.217900    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"]
Sep 18 21:13:34.253841 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.253808    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"]
Sep 18 21:13:34.254161 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.254135    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container" containerID="cri-o://9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9" gracePeriod=30
Sep 18 21:13:34.254161 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.254148    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kube-rbac-proxy" containerID="cri-o://56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d" gracePeriod=30
Sep 18 21:13:34.372826 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.372809    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp_520b3434-38c2-40f4-a837-1a892aa5455a/storage-initializer/1.log"
Sep 18 21:13:34.374975 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.374960    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:34.503535 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.503468    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/520b3434-38c2-40f4-a837-1a892aa5455a-proxy-tls\") pod \"520b3434-38c2-40f4-a837-1a892aa5455a\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") "
Sep 18 21:13:34.503535 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.503501    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-cabundle-cert\") pod \"520b3434-38c2-40f4-a837-1a892aa5455a\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") "
Sep 18 21:13:34.503535 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.503525    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxj46\" (UniqueName: \"kubernetes.io/projected/520b3434-38c2-40f4-a837-1a892aa5455a-kube-api-access-kxj46\") pod \"520b3434-38c2-40f4-a837-1a892aa5455a\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") "
Sep 18 21:13:34.503860 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.503573    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/520b3434-38c2-40f4-a837-1a892aa5455a-kserve-provision-location\") pod \"520b3434-38c2-40f4-a837-1a892aa5455a\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") "
Sep 18 21:13:34.503860 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.503595    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-kube-rbac-proxy-sar-config\") pod \"520b3434-38c2-40f4-a837-1a892aa5455a\" (UID: \"520b3434-38c2-40f4-a837-1a892aa5455a\") "
Sep 18 21:13:34.503860 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.503839    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520b3434-38c2-40f4-a837-1a892aa5455a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "520b3434-38c2-40f4-a837-1a892aa5455a" (UID: "520b3434-38c2-40f4-a837-1a892aa5455a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:13:34.503997 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.503866    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "520b3434-38c2-40f4-a837-1a892aa5455a" (UID: "520b3434-38c2-40f4-a837-1a892aa5455a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:13:34.503997 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.503976    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "520b3434-38c2-40f4-a837-1a892aa5455a" (UID: "520b3434-38c2-40f4-a837-1a892aa5455a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:13:34.505596 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.505575    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520b3434-38c2-40f4-a837-1a892aa5455a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "520b3434-38c2-40f4-a837-1a892aa5455a" (UID: "520b3434-38c2-40f4-a837-1a892aa5455a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:13:34.505647 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.505576    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520b3434-38c2-40f4-a837-1a892aa5455a-kube-api-access-kxj46" (OuterVolumeSpecName: "kube-api-access-kxj46") pod "520b3434-38c2-40f4-a837-1a892aa5455a" (UID: "520b3434-38c2-40f4-a837-1a892aa5455a"). InnerVolumeSpecName "kube-api-access-kxj46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:13:34.605120 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.605088    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/520b3434-38c2-40f4-a837-1a892aa5455a-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:34.605120 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.605114    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:34.605120 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.605123    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxj46\" (UniqueName: \"kubernetes.io/projected/520b3434-38c2-40f4-a837-1a892aa5455a-kube-api-access-kxj46\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:34.605328 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.605132    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/520b3434-38c2-40f4-a837-1a892aa5455a-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:34.605328 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.605141    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/520b3434-38c2-40f4-a837-1a892aa5455a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:34.683066 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.683026    2566 generic.go:356] "Generic (PLEG): container finished" podID="897f4c50-f820-43e9-971f-11836288f774" containerID="56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d" exitCode=2
Sep 18 21:13:34.683223 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.683089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" event={"ID":"897f4c50-f820-43e9-971f-11836288f774","Type":"ContainerDied","Data":"56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d"}
Sep 18 21:13:34.684329 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.684312    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp_520b3434-38c2-40f4-a837-1a892aa5455a/storage-initializer/1.log"
Sep 18 21:13:34.684421 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.684363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp" event={"ID":"520b3434-38c2-40f4-a837-1a892aa5455a","Type":"ContainerDied","Data":"5e6c723521b44f2cb86e6183c414e7935b6358c1e2a8c1af8b2eeebf2fb29df2"}
Sep 18 21:13:34.684421 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.684398    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"
Sep 18 21:13:34.684421 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.684410    2566 scope.go:117] "RemoveContainer" containerID="ddf512e703990ec1f123c96a5f22d21dcf32b369ea538b7ec5778ef5e5ea81c2"
Sep 18 21:13:34.718873 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.718851    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"]
Sep 18 21:13:34.721841 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:34.721820    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-51789d-predictor-6ccf5596f6-h6cmp"]
Sep 18 21:13:35.459159 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:35.459130    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="520b3434-38c2-40f4-a837-1a892aa5455a" path="/var/lib/kubelet/pods/520b3434-38c2-40f4-a837-1a892aa5455a/volumes"
Sep 18 21:13:38.694496 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.694474    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:13:38.703789 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.703762    2566 generic.go:356] "Generic (PLEG): container finished" podID="897f4c50-f820-43e9-971f-11836288f774" containerID="9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9" exitCode=0
Sep 18 21:13:38.703893 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.703810    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"
Sep 18 21:13:38.703893 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.703809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" event={"ID":"897f4c50-f820-43e9-971f-11836288f774","Type":"ContainerDied","Data":"9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9"}
Sep 18 21:13:38.703995 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.703940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5" event={"ID":"897f4c50-f820-43e9-971f-11836288f774","Type":"ContainerDied","Data":"db9ef66e6785d4ace252e4f2e10ce84078ce13f66bfb4e9e0a6fa39ad1894ce7"}
Sep 18 21:13:38.703995 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.703971    2566 scope.go:117] "RemoveContainer" containerID="56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d"
Sep 18 21:13:38.710970 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.710955    2566 scope.go:117] "RemoveContainer" containerID="9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9"
Sep 18 21:13:38.721700 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.721680    2566 scope.go:117] "RemoveContainer" containerID="d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35"
Sep 18 21:13:38.728903 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.728887    2566 scope.go:117] "RemoveContainer" containerID="56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d"
Sep 18 21:13:38.729203 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:38.729185    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d\": container with ID starting with 56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d not found: ID does not exist" containerID="56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d"
Sep 18 21:13:38.729294 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.729216    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d"} err="failed to get container status \"56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d\": rpc error: code = NotFound desc = could not find container \"56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d\": container with ID starting with 56dca60c3e24caaa895c9080e0e086e7fc79fb36a4802b1c2c3d49d6bc06de7d not found: ID does not exist"
Sep 18 21:13:38.729294 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.729240    2566 scope.go:117] "RemoveContainer" containerID="9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9"
Sep 18 21:13:38.729492 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:38.729477    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9\": container with ID starting with 9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9 not found: ID does not exist" containerID="9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9"
Sep 18 21:13:38.729531 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.729498    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9"} err="failed to get container status \"9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9\": rpc error: code = NotFound desc = could not find container \"9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9\": container with ID starting with 9232c85cd3a04a77229e9f86c551db8b9234ddbdf5f4573ae3b9f9f04b7fd2c9 not found: ID does not exist"
Sep 18 21:13:38.729531 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.729513    2566 scope.go:117] "RemoveContainer" containerID="d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35"
Sep 18 21:13:38.729719 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:13:38.729707    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35\": container with ID starting with d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35 not found: ID does not exist" containerID="d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35"
Sep 18 21:13:38.729767 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.729721    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35"} err="failed to get container status \"d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35\": rpc error: code = NotFound desc = could not find container \"d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35\": container with ID starting with d6ba4f13a7a2834a042bc5a05fc2695014c1adb328922644d4389614f7851a35 not found: ID does not exist"
Sep 18 21:13:38.836174 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.836151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897f4c50-f820-43e9-971f-11836288f774-kserve-provision-location\") pod \"897f4c50-f820-43e9-971f-11836288f774\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") "
Sep 18 21:13:38.836329 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.836196    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-kube-rbac-proxy-sar-config\") pod \"897f4c50-f820-43e9-971f-11836288f774\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") "
Sep 18 21:13:38.836329 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.836221    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/897f4c50-f820-43e9-971f-11836288f774-kube-api-access-6mpkt\") pod \"897f4c50-f820-43e9-971f-11836288f774\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") "
Sep 18 21:13:38.836329 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.836275    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-cabundle-cert\") pod \"897f4c50-f820-43e9-971f-11836288f774\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") "
Sep 18 21:13:38.836329 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.836293    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897f4c50-f820-43e9-971f-11836288f774-proxy-tls\") pod \"897f4c50-f820-43e9-971f-11836288f774\" (UID: \"897f4c50-f820-43e9-971f-11836288f774\") "
Sep 18 21:13:38.836555 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.836477    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897f4c50-f820-43e9-971f-11836288f774-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "897f4c50-f820-43e9-971f-11836288f774" (UID: "897f4c50-f820-43e9-971f-11836288f774"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:13:38.836608 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.836565    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "897f4c50-f820-43e9-971f-11836288f774" (UID: "897f4c50-f820-43e9-971f-11836288f774"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:13:38.836646 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.836609    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "897f4c50-f820-43e9-971f-11836288f774" (UID: "897f4c50-f820-43e9-971f-11836288f774"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:13:38.838216 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.838197    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897f4c50-f820-43e9-971f-11836288f774-kube-api-access-6mpkt" (OuterVolumeSpecName: "kube-api-access-6mpkt") pod "897f4c50-f820-43e9-971f-11836288f774" (UID: "897f4c50-f820-43e9-971f-11836288f774"). InnerVolumeSpecName "kube-api-access-6mpkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:13:38.838313 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.838295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897f4c50-f820-43e9-971f-11836288f774-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "897f4c50-f820-43e9-971f-11836288f774" (UID: "897f4c50-f820-43e9-971f-11836288f774"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:13:38.937268 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.937243    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897f4c50-f820-43e9-971f-11836288f774-kserve-provision-location\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:38.937268 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.937265    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:38.937404 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.937275    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mpkt\" (UniqueName: \"kubernetes.io/projected/897f4c50-f820-43e9-971f-11836288f774-kube-api-access-6mpkt\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:38.937404 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.937285    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897f4c50-f820-43e9-971f-11836288f774-cabundle-cert\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:38.937404 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:38.937295    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897f4c50-f820-43e9-971f-11836288f774-proxy-tls\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:13:39.025190 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:39.025164    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"]
Sep 18 21:13:39.029072 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:39.029052    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-51789d-predictor-7659f869cf-7nrf5"]
Sep 18 21:13:39.459467 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:39.459426    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="897f4c50-f820-43e9-971f-11836288f774" path="/var/lib/kubelet/pods/897f4c50-f820-43e9-971f-11836288f774/volumes"
Sep 18 21:13:40.547798 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.547766    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7bmlq/must-gather-xknlp"]
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548068    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548081    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548097    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548102    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548110    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kube-rbac-proxy"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548117    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kube-rbac-proxy"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548124    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="storage-initializer"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548129    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="storage-initializer"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548136    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520b3434-38c2-40f4-a837-1a892aa5455a" containerName="storage-initializer"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548141    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="520b3434-38c2-40f4-a837-1a892aa5455a" containerName="storage-initializer"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548147    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kube-rbac-proxy"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548152    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kube-rbac-proxy"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548158    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="storage-initializer"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548163    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="storage-initializer"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548168    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520b3434-38c2-40f4-a837-1a892aa5455a" containerName="storage-initializer"
Sep 18 21:13:40.548181 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548174    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="520b3434-38c2-40f4-a837-1a892aa5455a" containerName="storage-initializer"
Sep 18 21:13:40.548911 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548215    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kserve-container"
Sep 18 21:13:40.548911 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548222    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="520b3434-38c2-40f4-a837-1a892aa5455a" containerName="storage-initializer"
Sep 18 21:13:40.548911 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548230    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kube-rbac-proxy"
Sep 18 21:13:40.548911 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548236    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b49dc5d7-6299-4faa-9a14-331756d813ff" containerName="kserve-container"
Sep 18 21:13:40.548911 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548241    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="520b3434-38c2-40f4-a837-1a892aa5455a" containerName="storage-initializer"
Sep 18 21:13:40.548911 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.548248    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="897f4c50-f820-43e9-971f-11836288f774" containerName="kube-rbac-proxy"
Sep 18 21:13:40.552495 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.552478    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:13:40.554974 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.554954    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7bmlq\"/\"kube-root-ca.crt\""
Sep 18 21:13:40.556280 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.556152    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7bmlq\"/\"openshift-service-ca.crt\""
Sep 18 21:13:40.556280 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.556175    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7bmlq\"/\"default-dockercfg-5bk6b\""
Sep 18 21:13:40.558112 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.558088    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bmlq/must-gather-xknlp"]
Sep 18 21:13:40.648661 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.648627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhk9l\" (UniqueName: \"kubernetes.io/projected/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-kube-api-access-bhk9l\") pod \"must-gather-xknlp\" (UID: \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\") " pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:13:40.648833 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.648667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-must-gather-output\") pod \"must-gather-xknlp\" (UID: \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\") " pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:13:40.749521 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.749484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhk9l\" (UniqueName: \"kubernetes.io/projected/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-kube-api-access-bhk9l\") pod \"must-gather-xknlp\" (UID: \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\") " pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:13:40.749521 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.749519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-must-gather-output\") pod \"must-gather-xknlp\" (UID: \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\") " pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:13:40.749803 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.749788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-must-gather-output\") pod \"must-gather-xknlp\" (UID: \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\") " pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:13:40.757479 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.757448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhk9l\" (UniqueName: \"kubernetes.io/projected/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-kube-api-access-bhk9l\") pod \"must-gather-xknlp\" (UID: \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\") " pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:13:40.884120 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:40.884047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:13:41.150584 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:41.150561    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bmlq/must-gather-xknlp"]
Sep 18 21:13:41.152817 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:13:41.152788    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7b1cdf2_9177_4d8d_8fd8_42ac1e787546.slice/crio-a9062be801bb4a157dcf93394168e521225a4d04707baece7324c5a835ba89e0 WatchSource:0}: Error finding container a9062be801bb4a157dcf93394168e521225a4d04707baece7324c5a835ba89e0: Status 404 returned error can't find the container with id a9062be801bb4a157dcf93394168e521225a4d04707baece7324c5a835ba89e0
Sep 18 21:13:41.154493 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:41.154477    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:13:41.716806 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:41.716772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bmlq/must-gather-xknlp" event={"ID":"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546","Type":"ContainerStarted","Data":"a9062be801bb4a157dcf93394168e521225a4d04707baece7324c5a835ba89e0"}
Sep 18 21:13:47.747245 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:47.747155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bmlq/must-gather-xknlp" event={"ID":"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546","Type":"ContainerStarted","Data":"f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40"}
Sep 18 21:13:47.747245 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:47.747204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bmlq/must-gather-xknlp" event={"ID":"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546","Type":"ContainerStarted","Data":"b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187"}
Sep 18 21:13:47.763767 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:13:47.763705    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7bmlq/must-gather-xknlp" podStartSLOduration=1.5891933169999999 podStartE2EDuration="7.763686347s" totalImagesPullingTime="6.17449303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:13:47.762323227 +0000 UTC m=+3196.793249369" watchObservedRunningTime="2026-09-18 21:13:47.763686347 +0000 UTC m=+3196.794612488"
Sep 18 21:14:15.872554 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:15.872522    2566 generic.go:356] "Generic (PLEG): container finished" podID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" containerID="b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187" exitCode=0
Sep 18 21:14:15.872904 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:15.872561    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bmlq/must-gather-xknlp" event={"ID":"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546","Type":"ContainerDied","Data":"b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187"}
Sep 18 21:14:15.872904 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:15.872870    2566 scope.go:117] "RemoveContainer" containerID="b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187"
Sep 18 21:14:16.822393 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:16.822362    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7bmlq_must-gather-xknlp_b7b1cdf2-9177-4d8d-8fd8-42ac1e787546/gather/0.log"
Sep 18 21:14:17.557819 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.557785    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2nxhr/must-gather-rsv7c"]
Sep 18 21:14:17.560484 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.560462    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2nxhr/must-gather-rsv7c"
Sep 18 21:14:17.563123 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.563105    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2nxhr\"/\"kube-root-ca.crt\""
Sep 18 21:14:17.564225 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.564207    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2nxhr\"/\"default-dockercfg-n49wm\""
Sep 18 21:14:17.564279 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.564266    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2nxhr\"/\"openshift-service-ca.crt\""
Sep 18 21:14:17.571650 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.571628    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2nxhr/must-gather-rsv7c"]
Sep 18 21:14:17.655056 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.655027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnwx6\" (UniqueName: \"kubernetes.io/projected/7441b103-693b-49fe-b754-dac18aaac74f-kube-api-access-pnwx6\") pod \"must-gather-rsv7c\" (UID: \"7441b103-693b-49fe-b754-dac18aaac74f\") " pod="openshift-must-gather-2nxhr/must-gather-rsv7c"
Sep 18 21:14:17.655189 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.655066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7441b103-693b-49fe-b754-dac18aaac74f-must-gather-output\") pod \"must-gather-rsv7c\" (UID: \"7441b103-693b-49fe-b754-dac18aaac74f\") " pod="openshift-must-gather-2nxhr/must-gather-rsv7c"
Sep 18 21:14:17.756344 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.756317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnwx6\" (UniqueName: \"kubernetes.io/projected/7441b103-693b-49fe-b754-dac18aaac74f-kube-api-access-pnwx6\") pod \"must-gather-rsv7c\" (UID: \"7441b103-693b-49fe-b754-dac18aaac74f\") " pod="openshift-must-gather-2nxhr/must-gather-rsv7c"
Sep 18 21:14:17.756508 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.756354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7441b103-693b-49fe-b754-dac18aaac74f-must-gather-output\") pod \"must-gather-rsv7c\" (UID: \"7441b103-693b-49fe-b754-dac18aaac74f\") " pod="openshift-must-gather-2nxhr/must-gather-rsv7c"
Sep 18 21:14:17.756712 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.756694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7441b103-693b-49fe-b754-dac18aaac74f-must-gather-output\") pod \"must-gather-rsv7c\" (UID: \"7441b103-693b-49fe-b754-dac18aaac74f\") " pod="openshift-must-gather-2nxhr/must-gather-rsv7c"
Sep 18 21:14:17.764248 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.764228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnwx6\" (UniqueName: \"kubernetes.io/projected/7441b103-693b-49fe-b754-dac18aaac74f-kube-api-access-pnwx6\") pod \"must-gather-rsv7c\" (UID: \"7441b103-693b-49fe-b754-dac18aaac74f\") " pod="openshift-must-gather-2nxhr/must-gather-rsv7c"
Sep 18 21:14:17.869302 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:17.869238    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2nxhr/must-gather-rsv7c"
Sep 18 21:14:18.137136 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:18.136792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2nxhr/must-gather-rsv7c"]
Sep 18 21:14:18.139428 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:14:18.139397    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7441b103_693b_49fe_b754_dac18aaac74f.slice/crio-5bfeb3c5254662d5332d1591b9748c6eec728b0ff53d4e5052071cb17b34b952 WatchSource:0}: Error finding container 5bfeb3c5254662d5332d1591b9748c6eec728b0ff53d4e5052071cb17b34b952: Status 404 returned error can't find the container with id 5bfeb3c5254662d5332d1591b9748c6eec728b0ff53d4e5052071cb17b34b952
Sep 18 21:14:18.884480 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:18.884454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2nxhr/must-gather-rsv7c" event={"ID":"7441b103-693b-49fe-b754-dac18aaac74f","Type":"ContainerStarted","Data":"2da16257ba04ec37da21ede696e22b9893115e0d7e0a767a45791f4daebeeb41"}
Sep 18 21:14:18.884768 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:18.884499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2nxhr/must-gather-rsv7c" event={"ID":"7441b103-693b-49fe-b754-dac18aaac74f","Type":"ContainerStarted","Data":"5bfeb3c5254662d5332d1591b9748c6eec728b0ff53d4e5052071cb17b34b952"}
Sep 18 21:14:19.893464 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:19.893419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2nxhr/must-gather-rsv7c" event={"ID":"7441b103-693b-49fe-b754-dac18aaac74f","Type":"ContainerStarted","Data":"689ebd15e4a7a3e2337e837a2c4857be6047a6c279955401852285d15243d92d"}
Sep 18 21:14:19.909915 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:19.909856    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2nxhr/must-gather-rsv7c" podStartSLOduration=2.260358435 podStartE2EDuration="2.90983631s" totalImagesPullingTime="649.477875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:19.907830832 +0000 UTC m=+3228.938756974" watchObservedRunningTime="2026-09-18 21:14:19.90983631 +0000 UTC m=+3228.940762451"
Sep 18 21:14:20.402576 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:20.402383    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-msj7n_1a406b73-2c97-49f0-82b9-964afe466106/global-pull-secret-syncer/0.log"
Sep 18 21:14:20.497990 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:20.497916    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mh7w7_325de808-8e01-460b-9507-866c7181e24d/konnectivity-agent/0.log"
Sep 18 21:14:20.608214 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:20.608184    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-76.ec2.internal_36964665b9e807b88bd927b05ac588ed/haproxy/0.log"
Sep 18 21:14:22.385901 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.385847    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-7bmlq/must-gather-xknlp"]
Sep 18 21:14:22.386405 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.386141    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-7bmlq/must-gather-xknlp" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" containerName="copy" containerID="cri-o://f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40" gracePeriod=2
Sep 18 21:14:22.392505 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.390312    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-7bmlq/must-gather-xknlp"]
Sep 18 21:14:22.392505 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:14:22.390907    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xknlp\" is forbidden: User \"system:node:ip-10-0-143-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7bmlq\": no relationship found between node 'ip-10-0-143-76.ec2.internal' and this object" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:14:22.786323 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.786282    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7bmlq_must-gather-xknlp_b7b1cdf2-9177-4d8d-8fd8-42ac1e787546/copy/0.log"
Sep 18 21:14:22.790312 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.788613    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:14:22.798894 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:14:22.794573    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xknlp\" is forbidden: User \"system:node:ip-10-0-143-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7bmlq\": no relationship found between node 'ip-10-0-143-76.ec2.internal' and this object" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:14:22.906938 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.906878    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhk9l\" (UniqueName: \"kubernetes.io/projected/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-kube-api-access-bhk9l\") pod \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\" (UID: \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\") "
Sep 18 21:14:22.908577 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.908547    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-must-gather-output\") pod \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\" (UID: \"b7b1cdf2-9177-4d8d-8fd8-42ac1e787546\") "
Sep 18 21:14:22.910221 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.910192    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-kube-api-access-bhk9l" (OuterVolumeSpecName: "kube-api-access-bhk9l") pod "b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" (UID: "b7b1cdf2-9177-4d8d-8fd8-42ac1e787546"). InnerVolumeSpecName "kube-api-access-bhk9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:14:22.914877 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.914437    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7bmlq_must-gather-xknlp_b7b1cdf2-9177-4d8d-8fd8-42ac1e787546/copy/0.log"
Sep 18 21:14:22.914877 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.914833    2566 generic.go:356] "Generic (PLEG): container finished" podID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" containerID="f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40" exitCode=143
Sep 18 21:14:22.915061 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.914956    2566 scope.go:117] "RemoveContainer" containerID="f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40"
Sep 18 21:14:22.916608 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.915144    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:14:22.920159 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.918086    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" (UID: "b7b1cdf2-9177-4d8d-8fd8-42ac1e787546"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:14:22.920159 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:14:22.919353    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xknlp\" is forbidden: User \"system:node:ip-10-0-143-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7bmlq\": no relationship found between node 'ip-10-0-143-76.ec2.internal' and this object" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:14:22.929618 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.928636    2566 scope.go:117] "RemoveContainer" containerID="b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187"
Sep 18 21:14:22.967221 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.965937    2566 scope.go:117] "RemoveContainer" containerID="f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40"
Sep 18 21:14:22.967221 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:14:22.966387    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40\": container with ID starting with f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40 not found: ID does not exist" containerID="f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40"
Sep 18 21:14:22.967221 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.966422    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40"} err="failed to get container status \"f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40\": rpc error: code = NotFound desc = could not find container \"f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40\": container with ID starting with f51e5126ba80cca69ee862fcd4b3d279827e16c60534fdd6f3ca998b489f2a40 not found: ID does not exist"
Sep 18 21:14:22.967221 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.966448    2566 scope.go:117] "RemoveContainer" containerID="b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187"
Sep 18 21:14:22.967221 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:14:22.966774    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187\": container with ID starting with b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187 not found: ID does not exist" containerID="b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187"
Sep 18 21:14:22.967221 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:22.966822    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187"} err="failed to get container status \"b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187\": rpc error: code = NotFound desc = could not find container \"b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187\": container with ID starting with b7164aaa0c88391aee3fda590570c6aa8810c5abba73647ff4292e455f33f187 not found: ID does not exist"
Sep 18 21:14:23.013159 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:23.013057    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-must-gather-output\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:14:23.013159 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:23.013095    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhk9l\" (UniqueName: \"kubernetes.io/projected/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546-kube-api-access-bhk9l\") on node \"ip-10-0-143-76.ec2.internal\" DevicePath \"\""
Sep 18 21:14:23.230831 ip-10-0-143-76 kubenswrapper[2566]: E0918 21:14:23.230790    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xknlp\" is forbidden: User \"system:node:ip-10-0-143-76.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7bmlq\": no relationship found between node 'ip-10-0-143-76.ec2.internal' and this object" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" pod="openshift-must-gather-7bmlq/must-gather-xknlp"
Sep 18 21:14:23.462623 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:23.462588    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" path="/var/lib/kubelet/pods/b7b1cdf2-9177-4d8d-8fd8-42ac1e787546/volumes"
Sep 18 21:14:25.499007 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:25.498945    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8h7pv_1b507f6e-4807-4b73-aa93-942579bc41dc/node-exporter/0.log"
Sep 18 21:14:25.521893 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:25.521861    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8h7pv_1b507f6e-4807-4b73-aa93-942579bc41dc/kube-rbac-proxy/0.log"
Sep 18 21:14:25.558620 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:25.558593    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8h7pv_1b507f6e-4807-4b73-aa93-942579bc41dc/init-textfile/0.log"
Sep 18 21:14:25.855545 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:25.855506    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_38a506e5-80c7-4c05-8345-68a0cad94e5a/prometheus/0.log"
Sep 18 21:14:25.872153 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:25.872091    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_38a506e5-80c7-4c05-8345-68a0cad94e5a/config-reloader/0.log"
Sep 18 21:14:25.904996 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:25.904956    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_38a506e5-80c7-4c05-8345-68a0cad94e5a/thanos-sidecar/0.log"
Sep 18 21:14:25.965546 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:25.965514    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_38a506e5-80c7-4c05-8345-68a0cad94e5a/kube-rbac-proxy-web/0.log"
Sep 18 21:14:26.016573 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:26.016546    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_38a506e5-80c7-4c05-8345-68a0cad94e5a/kube-rbac-proxy/0.log"
Sep 18 21:14:26.039127 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:26.039099    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_38a506e5-80c7-4c05-8345-68a0cad94e5a/kube-rbac-proxy-thanos/0.log"
Sep 18 21:14:26.063311 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:26.063277    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_38a506e5-80c7-4c05-8345-68a0cad94e5a/init-config-reloader/0.log"
Sep 18 21:14:26.099079 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:26.098998    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-nmzkm_fddf80d0-1e99-4799-94e0-1415cbb122c4/prometheus-operator/0.log"
Sep 18 21:14:26.116564 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:26.116421    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-nmzkm_fddf80d0-1e99-4799-94e0-1415cbb122c4/kube-rbac-proxy/0.log"
Sep 18 21:14:26.142109 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:26.142034    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-hk6kj_fd2a4379-3d92-4f74-aabe-e5b380ea915d/prometheus-operator-admission-webhook/0.log"
Sep 18 21:14:28.680956 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:28.680917    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-5trl7_2f56239d-1c43-4f8d-905d-7f297ae92eeb/download-server/0.log"
Sep 18 21:14:29.440627 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.440591    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"]
Sep 18 21:14:29.441057 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.441040    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" containerName="copy"
Sep 18 21:14:29.441116 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.441061    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" containerName="copy"
Sep 18 21:14:29.441116 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.441076    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" containerName="gather"
Sep 18 21:14:29.441116 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.441085    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" containerName="gather"
Sep 18 21:14:29.441222 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.441168    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" containerName="copy"
Sep 18 21:14:29.441222 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.441180    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b1cdf2-9177-4d8d-8fd8-42ac1e787546" containerName="gather"
Sep 18 21:14:29.446293 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.446266    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.453902 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.453864    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"]
Sep 18 21:14:29.577145 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.577110    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-podres\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.577337 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.577175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-proc\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.577337 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.577225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-lib-modules\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.577337 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.577255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-sys\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.577337 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.577296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdzs\" (UniqueName: \"kubernetes.io/projected/fec2c6f3-c256-4865-bc7a-d5653550a0f0-kube-api-access-2tdzs\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.678531 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.678483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-proc\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.678731 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.678562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-lib-modules\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.678731 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.678592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-sys\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.678731 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.678635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdzs\" (UniqueName: \"kubernetes.io/projected/fec2c6f3-c256-4865-bc7a-d5653550a0f0-kube-api-access-2tdzs\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.678731 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.678690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-podres\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.678983 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.678856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-podres\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.678983 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.678923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-proc\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.679140 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.679005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-lib-modules\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.679140 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.679070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fec2c6f3-c256-4865-bc7a-d5653550a0f0-sys\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.688204 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.688173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdzs\" (UniqueName: \"kubernetes.io/projected/fec2c6f3-c256-4865-bc7a-d5653550a0f0-kube-api-access-2tdzs\") pod \"perf-node-gather-daemonset-k7j8w\" (UID: \"fec2c6f3-c256-4865-bc7a-d5653550a0f0\") " pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:29.760427 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:29.760343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:30.048096 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.048058    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"]
Sep 18 21:14:30.051357 ip-10-0-143-76 kubenswrapper[2566]: W0918 21:14:30.051325    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfec2c6f3_c256_4865_bc7a_d5653550a0f0.slice/crio-edfb9e320e94ebe0d06855a721d3fba03816a706233c3b8e71670c41e7fc95fe WatchSource:0}: Error finding container edfb9e320e94ebe0d06855a721d3fba03816a706233c3b8e71670c41e7fc95fe: Status 404 returned error can't find the container with id edfb9e320e94ebe0d06855a721d3fba03816a706233c3b8e71670c41e7fc95fe
Sep 18 21:14:30.137262 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.137236    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-phhxn_0c0cd081-7eaa-4ab3-beb0-48e17424b080/dns/0.log"
Sep 18 21:14:30.161119 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.161085    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-phhxn_0c0cd081-7eaa-4ab3-beb0-48e17424b080/kube-rbac-proxy/0.log"
Sep 18 21:14:30.184907 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.184888    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hglx4_d5396b35-d934-4cbd-bf7a-60e0ecb853f7/dns-node-resolver/0.log"
Sep 18 21:14:30.697458 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.697422    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b6c9fd4b9-6stdj_d5684f70-5e4f-41de-a375-5220cbb662b3/registry/0.log"
Sep 18 21:14:30.776263 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.776224    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z7w45_d8e15a8a-8986-42a3-9cba-945c5b6894a4/node-ca/0.log"
Sep 18 21:14:30.958892 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.958820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w" event={"ID":"fec2c6f3-c256-4865-bc7a-d5653550a0f0","Type":"ContainerStarted","Data":"b14c0941515bb1ec65795bf0a35466dcfadc805e0bfb26b9d6e647dfb9e95ea7"}
Sep 18 21:14:30.958892 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.958873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w" event={"ID":"fec2c6f3-c256-4865-bc7a-d5653550a0f0","Type":"ContainerStarted","Data":"edfb9e320e94ebe0d06855a721d3fba03816a706233c3b8e71670c41e7fc95fe"}
Sep 18 21:14:30.959092 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.958960    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:30.975938 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:30.975891    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w" podStartSLOduration=1.975876486 podStartE2EDuration="1.975876486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:30.974382619 +0000 UTC m=+3240.005308761" watchObservedRunningTime="2026-09-18 21:14:30.975876486 +0000 UTC m=+3240.006802643"
Sep 18 21:14:31.812050 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:31.812001    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dwvtg_82f87074-724c-4989-a468-5a5287a130e4/serve-healthcheck-canary/0.log"
Sep 18 21:14:32.329775 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:32.329747    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvxl9_85f0c548-516a-4413-b7f5-7012a0423ff1/kube-rbac-proxy/0.log"
Sep 18 21:14:32.348635 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:32.348605    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvxl9_85f0c548-516a-4413-b7f5-7012a0423ff1/exporter/0.log"
Sep 18 21:14:32.369316 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:32.369290    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvxl9_85f0c548-516a-4413-b7f5-7012a0423ff1/extractor/0.log"
Sep 18 21:14:34.713193 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:34.713166    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-vm9vw_83271296-0142-4900-86a5-ff5b7020ff8e/server/0.log"
Sep 18 21:14:35.374362 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:35.374329    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-4vrqz_012fc48d-f413-4e1f-90d5-341604a5854a/manager/0.log"
Sep 18 21:14:35.416635 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:35.416606    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6lpqh_b668aec6-09f6-4a9a-98ff-76536f37b802/s3-init/0.log"
Sep 18 21:14:35.432605 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:35.432583    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6lpqh_b668aec6-09f6-4a9a-98ff-76536f37b802/download-hf-model/0.log"
Sep 18 21:14:35.464714 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:35.464694    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-5dcw2_092b6d59-d454-41e9-9f87-703efabe4cd0/s3-tls-init-custom/0.log"
Sep 18 21:14:35.479349 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:35.479330    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-5dcw2_092b6d59-d454-41e9-9f87-703efabe4cd0/download-hf-model/0.log"
Sep 18 21:14:35.511505 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:35.511482    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4gd6t_e51312b7-3d96-4049-8a47-dd43fa91df57/s3-tls-init-serving/0.log"
Sep 18 21:14:35.526922 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:35.526902    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4gd6t_e51312b7-3d96-4049-8a47-dd43fa91df57/download-hf-model/0.log"
Sep 18 21:14:35.580328 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:35.580292    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-2zv9m_d0ce9076-49b4-4fda-a945-4c409458f061/seaweedfs-tls-custom/0.log"
Sep 18 21:14:36.976844 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:36.975943    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2nxhr/perf-node-gather-daemonset-k7j8w"
Sep 18 21:14:40.899397 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:40.899323    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2l4jx_b2f71b87-9bf3-454d-a2e1-97a0ffa97f52/kube-multus/0.log"
Sep 18 21:14:40.924317 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:40.924290    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8m8l_2c273cf8-0881-435f-be53-928e52dd5a58/kube-multus-additional-cni-plugins/0.log"
Sep 18 21:14:40.945829 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:40.945789    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8m8l_2c273cf8-0881-435f-be53-928e52dd5a58/egress-router-binary-copy/0.log"
Sep 18 21:14:40.969649 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:40.969623    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8m8l_2c273cf8-0881-435f-be53-928e52dd5a58/cni-plugins/0.log"
Sep 18 21:14:40.990662 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:40.990638    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8m8l_2c273cf8-0881-435f-be53-928e52dd5a58/bond-cni-plugin/0.log"
Sep 18 21:14:41.011265 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:41.011225    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8m8l_2c273cf8-0881-435f-be53-928e52dd5a58/routeoverride-cni/0.log"
Sep 18 21:14:41.058095 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:41.058056    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8m8l_2c273cf8-0881-435f-be53-928e52dd5a58/whereabouts-cni-bincopy/0.log"
Sep 18 21:14:41.074830 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:41.074798    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k8m8l_2c273cf8-0881-435f-be53-928e52dd5a58/whereabouts-cni/0.log"
Sep 18 21:14:41.637773 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:41.637686    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zfhl9_4e160c7e-4ea8-48e5-b932-32be44d1c289/network-metrics-daemon/0.log"
Sep 18 21:14:41.652804 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:41.652781    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zfhl9_4e160c7e-4ea8-48e5-b932-32be44d1c289/kube-rbac-proxy/0.log"
Sep 18 21:14:42.345548 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:42.345509    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-controller/0.log"
Sep 18 21:14:42.361838 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:42.361809    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/1.log"
Sep 18 21:14:42.362002 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:42.361852    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovn-acl-logging/0.log"
Sep 18 21:14:42.380930 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:42.380910    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/kube-rbac-proxy-node/0.log"
Sep 18 21:14:42.400897 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:42.400878    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 21:14:42.419202 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:42.419172    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/northd/0.log"
Sep 18 21:14:42.438329 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:42.438306    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/nbdb/0.log"
Sep 18 21:14:42.457874 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:42.457854    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/sbdb/0.log"
Sep 18 21:14:42.528985 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:42.528959    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7xnhp_ce0abf55-e019-4d4b-8bab-45921f3645c2/ovnkube-controller/0.log"
Sep 18 21:14:43.870170 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:43.870139    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-75k5k_01e6577b-6859-4947-9532-fc0b9edcf3cb/network-check-target-container/0.log"
Sep 18 21:14:44.812267 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:44.812233    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-b7b5n_7cb8b8b3-f9f1-41eb-b7d0-d571159c441a/iptables-alerter/0.log"
Sep 18 21:14:45.570940 ip-10-0-143-76 kubenswrapper[2566]: I0918 21:14:45.570912    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vzx2c_7def255f-b389-4260-b3bd-cb1ce422bef6/tuned/0.log"